Re: [jira] Commented: (MODPYTHON-55) Add a version attribute to the mod_python module.
Nicolas Lehuen wrote: It has not much impact since I simply don't use configure, nor make. To build under Windows XP, I just launch the Python script in dist\setup.py.in . Note the .in extension, which seems to mean that it should have been transformed by another script. However, when having a look at the script, I do not see any @@ macro. So I guess it's safe for it to be launched as is. The macro processing done by configure looks for any file ending in .in, makes the macro substitutions (anything enclosed by @@) and creates a new file with the .in extension stripped off. In fact it seems that we could solve all our versioning issues by tweaking this setup.py.in script. Looking at setup.py, there is already a function (getmp_version) which parses mpversion.h, so we're half way there. Just need to add another function to do dump the version string into __init__.py. Probably worth waiting for the patches from David Fraser though. See: http://issues.apache.org/jira/browse/MODPYTHON-55?page=comments#action_65861 Jim Regards, Nicolas 2005/5/20, Jim Gallacher [EMAIL PROTECTED]: Nicolas Lehuen wrote: The problem is that this won't work when building on Windows. I think we should use Python scripts instead of configure, grep et al which are not truly portable (well, I hear you, portable everywhere except on Windows :)). Use python? PYTHON Well, the very idea... :) Same principal, different tool. I'd still be willing to take a shot at it. Just curious - how does the use of awk and sed in configure impact building on Windows? Regards, Jim 2005/5/20, Jim Gallacher (JIRA) [EMAIL PROTECTED]: [ http://issues.apache.org/jira/browse/MODPYTHON-55?page=comments#action_65861 ] Jim Gallacher commented on MODPYTHON-55: Would it be appropriate to determine the version at configure time? If so we could create a file mod_python/__init__.py.in which would contain: version=@MP_VERSION@ and configure would grep src/include/mpversion.h for MPV_STRING and generate __init__.py from __init__.py.in. If this seems reasonable I can put something together. Add a version attribute to the mod_python module. - Key: MODPYTHON-55 URL: http://issues.apache.org/jira/browse/MODPYTHON-55 Project: mod_python Type: Improvement Versions: 3.1.4 Reporter: Nicolas Lehuen Assignee: Nicolas Lehuen Fix For: 3.2.0 (asked by dharana) There is no easy way to know the version of mod_python used from within an handler. Why not try to add a version string attribute to the mod_python module ? -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: FileSession segfaulting (Was: Re: _apache.emergency_unlock function?)
Hi dharana, I've been away for a few days, but should have some time to mess around with this today. dharana wrote: More info: It just happens with some classes. Some work and some won't, and I don't yet know why. Do you ever see segfaults if you only saving strings in your session, rather than objects? I have the feeling that I've seen mention of problems trying to pickle some objects. (Both DbmSession and FileSession create a pickle to store the data). I'm not sure if this was in the docs or on the mailing list. I'll dig around to see what I can find. Do you see the same segfault if you use DbmSession? If not then that would rule out a pickle problem. Some info: In my webapp when user logs in I save a User object (subclassing class Storable). And then I redirect. Works ok, no problem. Then, in another page I try to save a Website object (subclassing from both Storable and Website). It segfaults. Could it be something related to multiple inheritance? In my ignorance I thought so, but then when I try to save a Dbobj class just for testing purposes it also crashes. Dbobj inherits from object directly and has some methods defined as classmethod. My question is: how does mod_python knows where to look for a class while trying to retrieve it? Imagine I start a session in namespace a and then in namespace b I save a class instance which is only known to namespace b. What happens when namespace a tries to save the session or when it tries to load it again? How does it knows where to look for the class instance only known to namespace b? Interesting question. I'll need to refresh myself on the pickling mechanism. Maybe the problem lies there? Anyway I will save the object_id in the session instead of the class instance and I will load the object from it. I'm really sorry for the spam. My head starts spinning really fast when I see segfaults. Hey, this is not spam. :) And segfaults make my head spin too. Regards, Jim dharana wrote: This is driving me crazy. I'm finding it hard to make a simple testcase. I use a custom handler and somewhere in the code I do this: -- [snip] - from mypackage.mysubpackage.mysubsubpackage import db def run(req): req.sess['foo'] = object # req.sess['foo'] = db req.sess.save() return # I redirect the user another path with util.redirect and then the segfault # happens -- The moment I try to assign my own classes (ie: assigning db instead of object I must manually delete the pysid cookie to stop the segfaults. It seems as if mp corrupts or somehow leaves in an usable state the session after trying to save my objects. I tried with different objects and it happens with all of them. Does anyone knows how I can retrieve more info from the segfault? I don't like segfaults :( they don't tell why they don't like you. dharana wrote:
Re: [jira] Updated: (MODPYTHON-58) _apache._global_lock results in segfault when index number of mutexes
Hi Nicolas,
I've been working on the elusive request.get_session() session handling
mechanism and was trying to track down some issues with memory leaks and
segfaults (c extension code - yuck) when I spotted the problem with
_global_lock and it's siblings.
I don't think _global_lock will be used by developers unless they are
developing a new session class. The function is undocumented, and unless
people read the code for _apachemodule.c or Session.py they will never
even be aware of it's existence.
The use cases that I see are:
1. Acquire a system wide global lock, used to lock the dbmsession file
for example: _global_lock(server, None, index=0)
2. Acqurie a session lock: _global_lock(server, sid)
3. Debugging locking issues: _global_lock(server, None, index)
where -1 index # of mutexes.
As you suggest, we could use index = index % (glb-nlocks-1)+1) rather
than check that the index is in a proper range. However, since an index
!= 0 would only be used in case 3, I think this would just make
debugging more difficult.
I'm not worried about deadlocks if we use index % nlocks. Anybody that
tries to acquire more than one lock in the same request risks a deadlock
anyway, no matter how _global_lock is called.
Anyway, the point of this fix is to eliminate a bug, not add a feature.
:) I'm not stuck on any particular solution, as long as it doesn't segfault.
Regards,
Jim
Nicolas Lehuen wrote:
Him Jim,
If we add a test, then the developers will depend on the number of
mutexes that are configured. Why not do what is already done in
_apachemodule.c line 398, that is to say compute the given index
modulo the number of mutexes and use the resulting index ? I'm afraid
this might be deadlock-prone, but so is the already existing code in
that case... :(
Regards,
Nicolas
2005/6/1, Jim Gallacher (JIRA) [EMAIL PROTECTED]:
[ http://issues.apache.org/jira/browse/MODPYTHON-58?page=all ]
Jim Gallacher updated MODPYTHON-58:
---
Attachment: apachemodule.c-jg20050601-1.diff
Patch to fix issue is attached.
_apache._global_lock results in segfault when index number of mutexes
---
Key: MODPYTHON-58
URL: http://issues.apache.org/jira/browse/MODPYTHON-58
Project: mod_python
Type: Bug
Components: core
Versions: 3.1.4, 3.1.3, 3.2.0
Environment: All
Reporter: Jim Gallacher
Priority: Minor
Attachments: apachemodule.c-jg20050601-1.diff
All of the following calls will cause a segfault when the index is greater than
the number of global mutexes available or index -1.
eg. 32 mutexes created on apache startup
index = 100
_apache._global_lock(req.server, None, index)
_global_unlock(req.server, None, index)
_apache._global_trylock(req.server, None, index)
For all of the corresponding functions in _apachemodule.c, the value of index
is not checked before using it to access the contents of the global array of
mutex locks.
eg.
rv = apr_global_mutex_lock(glb-g_locks[index]);
I'll attach a patch for all three functions that does this check.
eg.
if ((index = (glb-nlocks)) || (index -1)) {
ap_log_error(APLOG_MARK, APLOG_WARNING, 0, s,
Index %d is out of range for number of global mutex
locks, index);
PyErr_SetString(PyExc_ValueError,
Lock index is out of range for number of global mutex
locks);
return NULL;
}
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: Managing and updating the web site
Nicolas Lehuen wrote: Grisha, can you tell us what is the process to release a new version, including the update of the web site ? Maybe include the web site source files in the subversion repository, so that we can edit it ? As for building the HTML documentation out of the LaTeX files, I'm a bit at a loss... Has anyone managed to do it ? Yes. Jim
Re: session handling - the next generation
Nicolas Lehuen wrote:
You should cast self to a requestobject* :
That worked. Running some tests.
Jim
((requestobject*)self)-session
Anyway, I'm currently rewriting the whole request_tp_traverse /
request_tp_clear / request_tp_clear functions like this :
static void request_tp_dealloc(requestobject *self)
{
// de-register the object from the GC
// before its deallocation, to prevent the
// GC to run on a partially de-allocated object
PyObject_GC_UnTrack(self);
request_tp_clear(self);
PyObject_GC_Del(self);
}
/**
** request_tp_traverse
**
*Traversal of the request object
*/
static int request_tp_traverse(requestobject* self, visitproc visit,
void *arg) {
int result;
if(self-dict) {result = visit(self-dict,arg); if(result) return result;}
if(self-connection) {result = visit(self-connection,arg);
if(result) return result;}
if(self-server) {result = visit(self-server,arg); if(result)
return result;}
if(self-next) {result = visit(self-next,arg); if(result) return result;}
if(self-prev) {result = visit(self-prev,arg); if(result) return result;}
if(self-main) {result = visit(self-main,arg); if(result) return result;}
if(self-headers_in) {result = visit(self-headers_in,arg);
if(result) return result;}
if(self-headers_out) {result = visit(self-headers_out,arg);
if(result) return result;}
if(self-err_headers_out) {result =
visit(self-err_headers_out,arg); if(result) return result;}
if(self-subprocess_env) {result =
visit(self-subprocess_env,arg); if(result) return result;}
if(self-notes) {result = visit(self-notes,arg); if(result) return result;}
if(self-phase) {result = visit(self-phase,arg); if(result) return result;}
// no need to Py_DECREF(dict) since the reference is borrowed
return 0;
}
static int request_tp_clear(requestobject *self)
{
PyObject* tmp;
tmp=self-dict; self-dict=NULL; Py_XDECREF(tmp);
tmp=self-connection; self-connection=NULL; Py_XDECREF(tmp);
tmp=self-server; self-server=NULL; Py_XDECREF(tmp);
tmp=self-next; self-next=NULL; Py_XDECREF(tmp);
tmp=self-prev; self-prev=NULL; Py_XDECREF(tmp);
tmp=self-main; self-main=NULL; Py_XDECREF(tmp);
tmp=self-headers_in; self-headers_in=NULL; Py_XDECREF(tmp);
tmp=self-headers_out; self-headers_out=NULL; Py_XDECREF(tmp);
tmp=self-err_headers_out; self-err_headers_out=NULL; Py_XDECREF(tmp);
tmp=self-subprocess_env; self-subprocess_env=NULL; Py_XDECREF(tmp);
tmp=self-notes; self-notes=NULL; Py_XDECREF(tmp);
tmp=self-phase; self-phase=NULL; Py_XDECREF(tmp);
tmp=self-hlo; self-hlo=NULL; Py_XDECREF(tmp);
tmp=self-dict; self-dict=NULL; Py_XDECREF(tmp);
tmp=self-dict; self-dict=NULL; Py_XDECREF(tmp);
tmp=self-dict; self-dict=NULL; Py_XDECREF(tmp);
tmp=self-dict; self-dict=NULL; Py_XDECREF(tmp);
return 0;
}
As you can guess from the source code, we'll have to add some code for
the session member when you'll integrate your code in subversion.
Regards,
Nicolas
2005/6/12, Jim Gallacher [EMAIL PROTECTED]:
Nicolas,
It fails to compile when I add your bit of code. Sure looks like it
should work though.
requestobject.c: In function `request_tp_traverse':
requestobject.c:1539: error: structure has no member named `session'
requestobject.c:1540: error: structure has no member named `session'
Just to be clear, it compiles and runs just fine without the extra check
added into request_tp_traverse.
Nicolas Lehuen wrote:
I'm re-reading the comment I wrote when I implemented tp_traverse :
// only traverse its dictionary since other fields defined in
request_rec_mbrs with type T_OBJECT
// cannot be the source of memory leaks (unless you really want it)
Turns out that we should at least traverse the next and prev in case
someone does something like req.next.my_prev = req. That's what I
meant by unless you really want it.
I wondered about that. :)
Jim
It's pretty difficult to get
there (you need an internal redirect), but you can. So I'm going to
have a look at re-implementing the tp_traverse handler.
Regards,
Nicolas
2005/6/12, Nicolas Lehuen [EMAIL PROTECTED]:
Duh, I get it. If you add a member to the request object, and this
member is not referenced in the request object's dictionary, then you
have to add a special case for it in the tp_traverse handler. In
requestobject.c :
/**
** request_tp_traverse
**
*Traversal of the request object
*/
static int request_tp_traverse(PyObject *self, visitproc visit, void *arg) {
PyObject *dict,*values,*item,*str;
int i,size;
// only traverse its dictionary since other fields defined in
request_rec_mbrs with type T_OBJECT
// cannot be the source of memory leaks (unless you really want it)
dict=*_PyObject_GetDictPtr(self);
if(dict) {
// this check is not needed, I guess, _PyObject_GetDictPtr
always give a pointer to a dict object.
if(PyDict_Check(dict)) {
i = visit(dict,arg
Re: session handling - the next generation
Nicolas Lehuen wrote:
Hi Jim,
Have you added the corresponding code in request_tp_clear and
request_tp_traverse ?
Yes.
I think you can check-in your code, since the debate about import
looks like it is going to take som time before the next release...
Will do.
Jim
Regards,
Nicolas
2005/6/12, Jim Gallacher [EMAIL PROTECTED]:
Nicolas Lehuen wrote:
Hi Jim,
After a few checks (unittest + load testing), I've checked in my
modifications ; you might want to update and merge it with your code.
I'm still getting a memory leak with the merged code. Should I commit my
changes anyway, or maybe we could create new svn branch for this testing?
Jim
Regards,
Nicolas
2005/6/12, Jim Gallacher [EMAIL PROTECTED]:
Nicolas Lehuen wrote:
You should cast self to a requestobject* :
That worked. Running some tests.
Jim
((requestobject*)self)-session
Anyway, I'm currently rewriting the whole request_tp_traverse /
request_tp_clear / request_tp_clear functions like this :
static void request_tp_dealloc(requestobject *self)
{
// de-register the object from the GC
// before its deallocation, to prevent the
// GC to run on a partially de-allocated object
PyObject_GC_UnTrack(self);
request_tp_clear(self);
PyObject_GC_Del(self);
}
/**
** request_tp_traverse
**
*Traversal of the request object
*/
static int request_tp_traverse(requestobject* self, visitproc visit,
void *arg) {
int result;
if(self-dict) {result = visit(self-dict,arg); if(result) return result;}
if(self-connection) {result = visit(self-connection,arg);
if(result) return result;}
if(self-server) {result = visit(self-server,arg); if(result)
return result;}
if(self-next) {result = visit(self-next,arg); if(result) return result;}
if(self-prev) {result = visit(self-prev,arg); if(result) return result;}
if(self-main) {result = visit(self-main,arg); if(result) return result;}
if(self-headers_in) {result = visit(self-headers_in,arg);
if(result) return result;}
if(self-headers_out) {result = visit(self-headers_out,arg);
if(result) return result;}
if(self-err_headers_out) {result =
visit(self-err_headers_out,arg); if(result) return result;}
if(self-subprocess_env) {result =
visit(self-subprocess_env,arg); if(result) return result;}
if(self-notes) {result = visit(self-notes,arg); if(result) return result;}
if(self-phase) {result = visit(self-phase,arg); if(result) return result;}
// no need to Py_DECREF(dict) since the reference is borrowed
return 0;
}
static int request_tp_clear(requestobject *self)
{
PyObject* tmp;
tmp=self-dict; self-dict=NULL; Py_XDECREF(tmp);
tmp=self-connection; self-connection=NULL; Py_XDECREF(tmp);
tmp=self-server; self-server=NULL; Py_XDECREF(tmp);
tmp=self-next; self-next=NULL; Py_XDECREF(tmp);
tmp=self-prev; self-prev=NULL; Py_XDECREF(tmp);
tmp=self-main; self-main=NULL; Py_XDECREF(tmp);
tmp=self-headers_in; self-headers_in=NULL; Py_XDECREF(tmp);
tmp=self-headers_out; self-headers_out=NULL; Py_XDECREF(tmp);
tmp=self-err_headers_out; self-err_headers_out=NULL; Py_XDECREF(tmp);
tmp=self-subprocess_env; self-subprocess_env=NULL; Py_XDECREF(tmp);
tmp=self-notes; self-notes=NULL; Py_XDECREF(tmp);
tmp=self-phase; self-phase=NULL; Py_XDECREF(tmp);
tmp=self-hlo; self-hlo=NULL; Py_XDECREF(tmp);
tmp=self-dict; self-dict=NULL; Py_XDECREF(tmp);
tmp=self-dict; self-dict=NULL; Py_XDECREF(tmp);
tmp=self-dict; self-dict=NULL; Py_XDECREF(tmp);
tmp=self-dict; self-dict=NULL; Py_XDECREF(tmp);
return 0;
}
As you can guess from the source code, we'll have to add some code for
the session member when you'll integrate your code in subversion.
Regards,
Nicolas
2005/6/12, Jim Gallacher [EMAIL PROTECTED]:
Nicolas,
It fails to compile when I add your bit of code. Sure looks like it
should work though.
requestobject.c: In function `request_tp_traverse':
requestobject.c:1539: error: structure has no member named `session'
requestobject.c:1540: error: structure has no member named `session'
Just to be clear, it compiles and runs just fine without the extra check
added into request_tp_traverse.
Nicolas Lehuen wrote:
I'm re-reading the comment I wrote when I implemented tp_traverse :
// only traverse its dictionary since other fields defined in
request_rec_mbrs with type T_OBJECT
// cannot be the source of memory leaks (unless you really want it)
Turns out that we should at least traverse the next and prev in case
someone does something like req.next.my_prev = req. That's what I
meant by unless you really want it.
I wondered about that. :)
Jim
It's pretty difficult to get
there (you need an internal redirect), but you can. So I'm going to
have a look at re-implementing the tp_traverse handler.
Regards,
Nicolas
2005/6/12, Nicolas Lehuen [EMAIL PROTECTED]:
Duh, I get it. If you add a member to the request object, and this
member is not referenced in the request object's dictionary, then you
have to add a special case
Re: PythonSessionOption - a new apache directive for session configuration
David Fraser wrote:
Hi Nicolas, Jim
PythonSessionCreate as suggested is nice for flexibility, but scary for
non-coders - I like Jim's multiple options for that reason
I would argue that PythonSessionCreate is less flexible for users and
admins. It requires a knowledge of the mod_python package structure
(Session.FileSession), which will then get hard coded into the apache
configuration files. If we decide in the future to split the Session
classes into separate files, we'll break all existing configurations.
PythonSessionOption will likely only be used to override the existing
defaults defined in the code, so most configs will look like this:
PythonSessionOption session DbmSession
PythonSessionOption timeout 28800
I'll grant you that the config file will become cluttered with a larger
number of PythonSessionOption directives for more a complicated setup,
but the same problem exists for PythonSessionCreate. eg.
PythonSessionCreate
Session.FileSession(session_directory='/var/lib/mod_python/sess',timeout=14400,secret='howdy',lock=1,fast_cleanup=1,verify_cleanup=1,graceperiod=20,this_string_is_hard_read=True)
Using python code in the config file means the admin needs to worry
about proper quoting of the string. We should not assume that the admin
will be a programmer. I can imagine the mailing list questions, and
Graham already spends enough time answering questions. :)
I also think that parsing the PythonSessionCreate string will turn out
to be a difficult task to do in a really robust way.
And the real reason for using PythonSessionOptions? Through the magic of
cut and paste programming I was able to adapt PythonOptions/get_options
in about 5 minutes with high confidence it would work the first time. It
did. :)
As far as refactoring Session.py, I've already done this to some extent
in my attempts to track down the memory leak with the req_get_session()
method - not a big job. Most of the config handling code would be added
to BaseSession.__init__(). My personal preference if we refactor
Session.py is to go all the way and split it into a package right now.
We are making enough changes in the way sessions are used that I think
it is the correct time, rather than half now and half later.
mod_python.session2 anyone? Session.py becomes wrapper around the new
package structure for backwards compatability, and coders using release
= 3.2.0 will use the PythonSessionOption, req.get_session() combo for
their new code.
Regards,
Jim
David
Nicolas Lehuen wrote:
Hi Jim,
Why not have a single 'PythonSessionCreate' option which contains the
line of code that should be evaled to create the session ?
Example :
PythonSessionCreate
Session.FileSession(session_directory='/var/lib/mod_python/sess',timeout=14400,lock=1)
The cost of eval could be brought down by precompiling the option
string to a code object and running the code object in precompiled
form.
This seems a bit more flexible to me and won't force you to refactor
the Session.py code.
Regards,
Nicolas
2005/6/12, Jim Gallacher [EMAIL PROTECTED]:
I've created a new apache directive called PythonSessionOption. This
would be used to configure session handling in the apache config file.
This data is accessed with a new request method,
req.get_session_options().
Although we could use the PythonOption directive instead of creating a
new one, I believe it's better to keep the session config data separate
so we don't need to worry about collisions with current user code or
configuration.
Typical Usage
-
In a test script mptest.py
def handler(req)
opts = req.get_session_options()
for k in sess_conf:
req.write('%s: %s' % (k,opts[k])
In Session.FileSession:
__init__(self,req,sid):
opts = req.get_session_options()
timeout = int(opts.get('timeout', DFT_TIMEOUT))
In an Apache config file:
VirtualHost 192.168.1.12:80
ServerAdmin [EMAIL PROTECTED]
ServerName example.com
DocumentRoot /var/www/
PythonSessionOption session FileSession
PythonSessionOption session_directory /var/lib/mod_python/sess
PythonSessionOption timeout 14400
PythonSessionOption lock 1
...
/VirtualHost
If there are no objections I'll commit the code. I have not refactored
Sessions.py to use the new configuration scheme just yet.
Regards,
Jim
Re: Session Benchmarks
Gregory (Grisha) Trubetskoy wrote: On Fri, 17 Jun 2005, Jim Gallacher wrote: I was thinking we'd still use the current global locking scheme, but keep the file open between requests. Not sure if this would be robust or just asking for dbm file corruption though. I'm pretty sure it won't work, I think you cannot have a dbm open by more than one process without getting it corrupted. Well of course not. Just a little brain fart on my part. Jim
Re: Session Benchmarks
Gregory (Grisha) Trubetskoy wrote: On Fri, 17 Jun 2005, Jim Gallacher wrote: Any objection to just a SqlSession base class? May be - it depends on how complex it becomes. Any attempts I've to generalize SQL/DB stuff tend to become a mess since there are no firm standards in this area, so this just may be something for the contrib (or whatever it ends up being), like Nicolas pointed out. I'm a little concerned about staying focused on closing the last bugs so that we get to a point where a release can be made, since there hasn't been one in such a long time... I was not actually proposing a SqlSession class for 3.2.0. I just want to finish off the last couple of issues of FileSession for this release, and catalog the SqlSession idea as a possible future feature. It really would be helpful if everyone could read minds. Maybe we could put a mind-reading feature in 3.3? :) Although that might push 3.3's release back to 2020. And could it be made thread safe... Regards, Jim
Re: Session Benchmarks
Nick wrote: Jim Gallacher wrote: Using bsddb3 would introduce new dependency for mod_python, so I don't know if it's a good idea to use transaction handling by default for DbmSession. Maybe we could offer a subclass? Starting with Python 2.3 this module is included in the standard python distribution as its bsddb module. Are you sure? The docs on python.org indicate otherwise. It would be sweet if bsddb3 was the std module. Jim
Re: Session Benchmarks
Nick wrote: Jim Gallacher wrote: Nick wrote: Jim Gallacher wrote: Using bsddb3 would introduce new dependency for mod_python, so I don't know if it's a good idea to use transaction handling by default for DbmSession. Maybe we could offer a subclass? Starting with Python 2.3 this module is included in the standard python distribution as its bsddb module. Are you sure? The docs on python.org indicate otherwise. It would be sweet if bsddb3 was the std module. Jim Directly from the Python module documentation: Starting with Python 2.3 the bsddb module requires the Berkeley DB library version 3.2 or later (it is known to work with 3.2 through 4.3 at the time of this writing). I saw that but did not parse it correctly. I just saw 'Legacy Interface' and assumed you would need to grab a different module for the current interface. Whereas the python documention actually says is: The following is a description of the legacy bsddb interface compatible with the old python bsddb module. For details about the more modern Db and DbEnv object oriented interface see the above mentioned pybsddb URL. Mea culpa. Jim
Re: Potential deadlock in psp.py
Nicolas Lehuen wrote: Hi Jim, Until now, we suspected that the way global locks are handled could be deadlock prone. You have just proved it. I know that global locks are expensive on some systems, especially if we want to use them in a multiprocess (forked) environment. That's why we are forced to have such a small pool of global locks. On the other hand, in a multithreaded environment, locks which are valid for the whole process are not so expensive ; indeed, we can create a whole bunch without bringing down the system (think about Java where all object potentially have a monitor which is equivalent to a lock). I think this is a strong incentive to abandon the forked model and go for the multi-threaded model (or the mono-threaded, asynchronous one). For concurrency control, the forked model does not scale, apparently. It would make life easier I suppose, but we are stuck with the fact that apache provides several different mpm models that need to be supported. Jim
Re: [jira] Commented: (MODPYTHON-59) Add get_session() method torequest object
Graham Dumpleton wrote: I can see two problems here. The first is that if the target of the internal redirect is a part of the URL namespace which is under the control of a different handler, or where ApplicationPath option was set explicitly to be different, the PYSID would potentially override a valid pysid for the alternate SID context provided by the browser. I get your point but is this actually a problem? Since it's an internal redirect, the browser will not be aware it is being redirected, so it will always send the same PYSID cookie. (Did that sentence make sense?) Even if the second handler sets a session cookie with a path for URL2, the browser will always send the cookie corresponding to URL1. Probably true. There may be convoluted cases where if someone was doing extra wierd stuff with overriding ApplicationPath and playing with the option being discussed for SessionCookieName, strange things might occur, but if people get that convoluted they get what they deserve. Agreed. BTW, Nicolas has already implemented a PythonOption to specify the cookie name. One other aspect of this that I have been thinking about is the feature in there for auto saving and unlocking a session before doing an internal redirect. In doing this you have changed how things work now and it does open up slim chance that existing code could break. Imagine where someone had stored stuff in the session object but then used internal redirect to force a special error response by triggering a handler outside of the session context. They didn't save the session because they wanted to discard their progressive changes. Now that it does an autosave, it may save stuff up code given that people didn't want it saved. Point taken. Also, you don't recreate the session object after internal redirect has returned. Since internal redirect returns control back to the caller still, there is a chance that someone might be expecting to still be able to use the session object in some way after the internal redirect has returned. Except for the session unlock/save, internal_redirect does not touch the session object. The session is still available to the caller, so there is no need to recreate it. Or am I missing something? Is my memory that internal redirect does return actually right? Yes. I thus question whether autosave on redirect is a good idea or not. People can explicitly do it now themselves. If the code is in there, it should perhaps at least be off by default and require an explicit option to turn it on. If it has to be turned on though, people may as well code it themselves. I've gone back and forth on the autosave. At this point I agree it's best to leave the saving to the user's code, so I'll take it out. At one point I raised the possibility of having an autosave for sessions themselves and people were pretty much against that idea. By extension internal_redirect shouldn't save the session so we are at least consistent. I think the automatic session unlock needs to stay. It's just too easy for the user to forget a manual unlock, deadlock the session and hose their server in very short order. Regards, Jim
Re: 3.2
Gregory (Grisha) Trubetskoy wrote: On Thu, 28 Jul 2005, Nicolas Lehuen wrote: Note that there are 29 unscheduled issues : http://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=truemode=hidesorter/order=DESCsorter/field=priorityresolutionIds=-1pid=10640fixfor=-1 Maybe some of them should be included in the 3.2 release ? My inclanation is to just release whatever we have, and mark it as a beta release. The last true release was 3.1.3 in Feb 2004, which makes it 18 months (if my math is correct) Grisha I've either commited fixes or have fixes ready for 6 or 8 of those issues. Also there some that don't apply to 3.2 (eg website or mailing list issues). Must run right now but will make a list this evening of issues which can be closed. Jim
Re: 3.2
Gregory (Grisha) Trubetskoy wrote: On Thu, 28 Jul 2005, Jim Gallacher wrote: I've either commited fixes or have fixes ready for 6 or 8 of those issues. Also there some that don't apply to 3.2 (eg website or mailing list issues). Must run right now but will make a list this evening of issues which can be closed. Do you think you'll be able to create a tgz file? It would really be fantastic if someone other than me could do it. Grisha Yes I can do that. I've already done a couple of dry runs with good success. Jim
Re: Different approach to req.get_session().
Just so everyone is clear, implementation of req.get_session() or its equivalent has been deferred to version 3.3. Graham Dumpleton wrote: Some will know that I haven't been too keen on the way in which the proposed new req.get_session() method was being implemented. My concerns were that it was a magic box and didn't allow access to the underlying variable holding the session object so as to allow checks such as determining whether a session already existed before creating it. I also had my concerns about how the session would be automatically unlocked when req.internal_redirect() was used, something which may introduce backward compatibility problems. Anyway, have been thinking about this stuff on and off and have come with an idea which might be worth pursuing. Previously when discussing the issue of the req object not being available to apache.import_module() unless explicitly provided, and the implications of this as far as ensuring that settings for logging and autoreload were consistent between code and Apache configuration files, I proposed the idea of a request cache. http://www.mail-archive.com/python-dev@httpd.apache.org/msg00197.html This was where the request object was put in a globally accessible dictionary indexed by thread identifier. Having been done, the request object could be accessed from anywhere. There were various issues related to use of req.internal_request() that had to be catered for, but in short it acted like a thread specific global variable. Now consider something similar for the instance of a session object that is created. Ie., def Session(...): Determine session id if not supplied as argument. Check session object cache to see if session instance already stored for (threadId,sessionId). If cached session object exists, use it and return it. Otherwise create new session object, store it in the cache under the key (threadId,sessionId) and return it. A cleanup handler is already registered for a session object when a session object is created. This same cleanup handler can remove the session object from the cache. In respect of req.internal_redirect(), with the above there doesn't need to be any special code which unlocks the session. Instead, when any code executed as a consequence of req.internal_redirect() calls Session.Session() to create the session object, it will be given the already constructed session object instead of creating a new one. The session lock will never have been released and the code will simply use it. The code called by req.internal_redirect() can save the session, or it could even be done by the code which called req.internal_redirect() in the first place. Because the session instance is stored in a global cache within the session code and not in the request object itself, one can probably even totally do away with the req.get_session() method. Code will simply continue to use Session.Session() method and users can store it wherever they like. Well, thats the basic idea. It obviously needs a bit of work on the caching with respect to thread locks etc, but that shouldn't be too much work. Comments? Have I explained it well enough? Graham I think this may have merit although I need to mull it over. Would the session cache be on disk or in memory? If in memory, what are the implications in an mpm-prefork environment? If on disk, what are the performance implications? Will it introduce new and exciting locking issues for accessing the cache? I'm not looking for answers to the above at this point, just recording them for the record. Jim
Re: [jira] Updated: (MODPYTHON-69) Potential deadlock in psp cache
I've committed the fix. For some reason JIRA is picking up the subversion commits but not forwarding the message to the mailing list. This issue can be closed. Jim Jim Gallacher (JIRA) wrote: [ http://issues.apache.org/jira/browse/MODPYTHON-69?page=all ] Jim Gallacher updated MODPYTHON-69: --- Description: This issue was discussed on the python-dev mailing list but not followed up on. Fixing that now. In psp.py def dbm_cache_store(srv, dbmfile, filename, mtime, val): dbm_type = dbm_cache_type(dbmfile) ### potential deadlock here! ### _apache._global_lock(srv, pspcache) try: dbm = dbm_type.open(dbmfile, 'c') dbm[filename] = %d %s % (mtime, code2str(val)) finally: try: dbm.close() except: pass _apache._global_unlock(srv, pspcache) pspcache will hash to one of 31 mutexes. Therefore there is a 1 in 31 chance for a hash collision if a session is used in the same request, which would result in a deadlock. (This has been confirmed by testing.) Most obvious solution is to use the global lock 0, which will serialize all accesses to either pspcache.dbm. Global lock 0 is also used by DbmSession, but since the lock is not held for the duration of the request there should not be any additional deadlock issues. The fix is to replace the _apache._global_lock(srv, pspcache) with _apache._global_lock(srv, None, 0) The corresponding lock handling in dbm_cache_get() will also need the same fix. I will commit the this fix shortly. was: This issue was discussed on the python-dev mailing list but not followed up on. Fixing that now. In psp.py def dbm_cache_store(srv, dbmfile, filename, mtime, val): dbm_type = dbm_cache_type(dbmfile) ### potential deadlock here! ### _apache._global_lock(srv, pspcache) try: dbm = dbm_type.open(dbmfile, 'c') dbm[filename] = %d %s % (mtime, code2str(val)) finally: try: dbm.close() except: pass _apache._global_unlock(srv, pspcache) pspcache will hash to one of 31 mutexes. Therefore there is a 1 in 31 chance for a hash collision if a session is used in the same request, which would result in a deadlock. (This has been confirmed by testing.) Most obvious solution is to use the global lock 0, which will serialize all accesses to either pspcache.dbm. Global lock 0 is also used by DbmSession, but since the lock is not held for the duration of the request there should not be any additional deadlock issues. The fix is to replace the _apache._global_lock(srv, pspcache) with _apache._global_lock(srv, None, 0) I will commit the this fix shortly. Potential deadlock in psp cache --- Key: MODPYTHON-69 URL: http://issues.apache.org/jira/browse/MODPYTHON-69 Project: mod_python Type: Bug Components: publisher Versions: 3.2.0 Environment: All Reporter: Jim Gallacher This issue was discussed on the python-dev mailing list but not followed up on. Fixing that now. In psp.py def dbm_cache_store(srv, dbmfile, filename, mtime, val): dbm_type = dbm_cache_type(dbmfile) ### potential deadlock here! ### _apache._global_lock(srv, pspcache) try: dbm = dbm_type.open(dbmfile, 'c') dbm[filename] = %d %s % (mtime, code2str(val)) finally: try: dbm.close() except: pass _apache._global_unlock(srv, pspcache) pspcache will hash to one of 31 mutexes. Therefore there is a 1 in 31 chance for a hash collision if a session is used in the same request, which would result in a deadlock. (This has been confirmed by testing.) Most obvious solution is to use the global lock 0, which will serialize all accesses to either pspcache.dbm. Global lock 0 is also used by DbmSession, but since the lock is not held for the duration of the request there should not be any additional deadlock issues. The fix is to replace the _apache._global_lock(srv, pspcache) with _apache._global_lock(srv, None, 0) The corresponding lock handling in dbm_cache_get() will also need the same fix. I will commit the this fix shortly.
Re: _apache._global_lock theory
Daniel Popowich wrote: Jim Gallacher writes: Daniel Popowich wrote: The recent discussion of max locks and deadlocking issues with _apache._global_(un)?lock() are timely for me: I'm in the middle of writing a caching module for mod_python servlets so a developer can have the output of a servlet cached, keyed on the hash of the URI, for future requests. The goal, of course, is to increase throughput for dynamic pages with long shelf life (e.g., content manager updates database once per day, so the HTML only needs to be generated once per day, not per request). I need locking. My first gut instinct was to go to the fcntl module (lockf), but this is only available for unix. My second gut instinct was: so what? :-) Wanting a true x-platform tool I then thought of mod_python.Session needing locks, poked around the code, and saw how it uses _apache._global_lock(). Further poking around showed me how psp code caching uses the function. About the time I began worrying about deadlock issues I found the thread on this list discussing the same problems. The solution for Session and psp code caching is to explicitly use lock id 0. This works as long as a module does not hold the lock for the whole request, but unlocks immediately after acquiring the needed resource. Fine. Just to be clear, sessions use the locks above index 0 for session locking. The session id is hashed to determine which index is used. DbmSession uses lock index 0 to lock the dbm file for reading and writing the persistent session data. This is independent of the session lock. Right, not sessions proper, but by the mod_python.Session module for dbm file reading. So, my question: is this the recommended way for mod_python framework developers to acquire x-platform global locks? Explicitly use lock id 0? If so, is this a secret or should it be documented? I don't know if it's recommended, but I don't see a problem as long as the lock is held briefly and you make sure you unlock it when you are done. I suspect it is undocumented because it was never documented as opposed to some larger conspiracy. I guess I was too tongue-in-cheek...my question is: Is it not documented on *purpose*? Perhaps it should be documented for internal developers and framework developers? No, actually I understood your cheek. I was just too lazy to put in a smiley after my comment. I shall correct that now. :) And a winkey for good measure. ;) I used another cross platform approach in filesession_cleanup() in Session.py. I wanted to make sure only one request at a time was running the cleanup, and used the os.open() call to exclusively open a guard file. (OK, not a guard file, but my brain just went blank. Hopefully you get the idea.) I'm with ya... :-) Here is a code snippet: Thanks for the code...maybe I'll try both (your code and _apache._global_lock()) and benchmark my caching code with ab. Thinking out loud here...wouldn't it be good for mod_python to provide a facility for global locking based on some key? By default, the lock is per interpreter, but optionally per server? Given the oddities of python programming within an apache environment, especially a prefork MPM environment, it seems it would be a most valueable service. The Session, psp and 3rd-party locking (e.g. mpservlets) could all share the same code. That discussion will have to wait for another time. Time to call it a day. Regards, Jim
mod_python package maintainers - are you out there?
Nicolas Lehuen wrote: Another remark : has anyone suscribed to redhat, debian etc. mailing list to watch for such patches ? Not me. I don't understand why those guys aren't posting their patches on the mod_python mailing list. I was wondering the same thing. What would be better for us, subscribing to a bunch of mailing lists or contacting the maintainers directly and telling them we are interested in having them forward their bug reports and patches (and then hoping they do)? Having some contact with them directly is probably a good idea anyway. Subscribing to a bunch of mailing lists could result in a lot of uninteresting mail. ;) But first let's have a show of hands. How many people monitoring either of the mod_python or python-dev mailing lists are package maintainers for a distribution? Regards, Jim
Re: 3.2 (import and publisher issues)
Nicolas Lehuen wrote: MODPYTHON-34 has been fixed in the current version of the publisher, with the new importing system. As I've written before, I can roll back the part regarding the import system if you really want that, all the while maintaining the fix for MODPYTHON-34. I haven't had much to say on this topic (because I don't have a solid grasp of all the implications), but it seems like it's a real stumbling block for the release. I'd say if the current changes won't break any current code, or cause users unexpected behaviour, leave them in. Otherwise roll them back. It's better for the user to have to deal with old, documentented issues rather than discover brand new ones in their code, only to find we've changed the behaviour again in 3.3. Whichever route we take, we will obviously work on this further in 3.3 but for now let's get 3.2.0b out the door. Regards, Jim
Re: 3.2 (import and publisher issues)
Nicolas Lehuen wrote: I'd like to stress the fact that a lot of issues currently in JIRA are related to the publisher. We have worked on it to solve some of these issues. I have made sure that both the old version and the new version pass a series of unit tests. I can't be sure that those unit tests reflect the whole range of usage patterns people could have of the publisher, but anyway, it's better than nothing. So, I think we should move forward on the 3.2 release. +1 The new publisher code is meant to make it work better, not worse, while retaining the compatibility with the current code. It's not a new publisher, it's a set of bug fixes. I mean, what is the purpose of releasing a new version of mod_python if we don't fix the dozen of bugs that are related to the publisher ? Regards, Nicolas 2005/8/10, Jim Gallacher [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: Nicolas Lehuen wrote: MODPYTHON-34 has been fixed in the current version of the publisher, with the new importing system. As I've written before, I can roll back the part regarding the import system if you really want that, all the while maintaining the fix for MODPYTHON-34. I haven't had much to say on this topic (because I don't have a solid grasp of all the implications), but it seems like it's a real stumbling block for the release. I'd say if the current changes won't break any current code, or cause users unexpected behaviour, leave them in. Otherwise roll them back. It's better for the user to have to deal with old, documentented issues rather than discover brand new ones in their code, only to find we've changed the behaviour again in 3.3. Whichever route we take, we will obviously work on this further in 3.3 but for now let's get 3.2.0b out the door. Regards, Jim
Re: 3.2 Compile problems with gcc 4.0
Gregory (Grisha) Trubetskoy wrote: On Wed, 10 Aug 2005, Jim Gallacher wrote: Compilation fails with the following: requestobject.c: In function 'request_tp_dealloc': requestobject.c:1482: warning: implicit declaration of function 'request_tp_clear' This looks like a bug - I guess GCC 3 defaulted to static for implicitely declared functions, or perhaps didn't mind a non-static declaration followed by a static declaration. gcc 3.3 just generates a warning: requestobject.c: In function `request_tp_dealloc': requestobject.c:1479: warning: implicit declaration of function `request_tp_clear' requestobject.c: At top level: requestobject.c:1511: warning: `request_tp_clear' was declared implicitly `extern' and later `static' requestobject.c:1479: warning: previous declaration of `request_tp_clear' As a sidenote - (wearing my dictator hat) - we really should be following the Apache coding style - having no spaces after if and several statements per line makes it difficult to read. Also, repetitive code is probably a good case for using an extra function or a macro (macro's should be last resort though because they can be hard to read and debug - but in this particular case a macro may be just fine) OK, chief. :) I don't mind doing an audit for Apache coding style compliance, but in a post 3.2.0 time frame. Let's not start mucking with working code just before the release. It's easy enough to fix (and I will) by adding a function prototype for request_tp_clear before it is called in request_tp_dealloc(). eg. static int request_tp_clear(requestobject *self); I think it can be fixed by simply moving tp_dealloc after tp_clear. This is true, but I wasn't sure if it that was the preferred way. I've noticed that in general functions are implicitly declared in mod_python. Is there a technical reason for this, or is it just one of those things? Not really - implicit declaration is when something is referenced before being declared which in most cases results in compilation errors. What you probably mean to say is that they do not have forward declarations, but you generally don't need them if code is ordered correctly - there is no reason to have a forward declaration for every function. OK. I'll fix the bug by moving request_tp_clear ahead of request_tp_dealloc. Jim
Re: 3.2 beta release today?
Gregory (Grisha) Trubetskoy wrote: On Mon, 15 Aug 2005, Graham Dumpleton wrote: Does this imply no more fixes for outstanding problems even though the necessary patches have been posted up on JIRA. Eg,. If we stick to the schedule, then yes. http://issues.apache.org/jira/browse/MODPYTHON-12 The patch for this one would have eliminated the last workaround that I keep having to propagate in my code, even though the minimum requirement for my code was going to be to have 3.2. Guess I'll have to go back and put the workaround in again until 3.2.X. :-( But you have an opportunity to vote -1 on the new tarball because of this issue (the rules are that anyone on this list can vote). It does not look like a major patch. The diff is only a few lines. I can commit it as the last act before shoving the beta out the door today. Jim
Re: flex [was mod_python 3.2.0-BETA available for testing]
Gregory (Grisha) Trubetskoy wrote: OK, here is the flex scoop - as the the docs point out, anything before 2.5.31 is not reentrant and I think even uses a slightly different interface so older flex won't even process the psp_parser.l file correctly. Looking at Fedora Core 4, it still has flex 2.5.4a. (Note that 2.5.31 2.5.4 because 31 4 - I for a while had trouble seeing that for some reason), so the new flex is still not commonplace. This is true. I took a quick survey when I was adding the --with-flex option, and 2.5.4 was the most common. Oh, and the version numbering confused me for a while as well, but the configure script does take it into account when deciding if it has the correct flex. So until reentrant flex becomes commonplace, the solution was to include a pre-parsed psp_parser.c so that you woudln't need flex at all to compile mod_python. Looks like this still should be the case. The ./configure should just print a warning that if flex is not found or too old, should you need to rebuild psp_parser.c you will need to get the right version of flex. When I added the flex detection I had not considered the case where there was *no* flex on the system... just tested for the correct version. :-( Also, I wasn't sure how verbose to make the configure message. I just figured that people would see the WARNING and then go to the README to see what the deal was, which would have been the perfect plan... if I had actually updated the README accordingly. I'll make the fixes to configure and the README. Is the explanation in doc-html/inst-configure.html clear or does it need to be revised as well? Jim See doc-html/inst-configure.html: Attempts to locate flex and determine its version. If flex cannot be found in your PATH configure will fail. If the wrong version is found configure will generate a warning. You can generally ignore this warning unless you need to re-create src/psp_parser.c. The parser used by psp (See 4.9) is written in C generated using flex. This requires a reentrant version of flex which at this time is 2.5.31. Most platforms however ship with version 2.5.4 which is not suitable, so a pre-generated copy of psp_parser.c is included with the source. If you do need to compile src/psp_parser.c you must get the correct flex version. If the first flex binary in the path is not suitable or not the one desired you can specify an alternative location with the --with-flex option, e.g: $ ./configure --with-flex=/usr/local/bin/flex I'll copy this into the README as well. In 3.1.x the path to flex was hard coded to /usr/local/bin, which caused compilation to fail and required the user to edit the Makefile manually. Other than the detection of the flex path, the behaviour is the same as 3.1.x. The previous warning about needing the correct flex version was just a comment in the Makefile. Any suggestions on how to make this clearer are welcome.
Re: flex [was mod_python 3.2.0-BETA available for testing]
Gregory (Grisha) Trubetskoy wrote: OK, here is the flex scoop - as the the docs point out, anything before 2.5.31 is not reentrant and I think even uses a slightly different interface so older flex won't even process the psp_parser.l file correctly. Looking at Fedora Core 4, it still has flex 2.5.4a. (Note that 2.5.31 2.5.4 because 31 4 - I for a while had trouble seeing that for some reason), so the new flex is still not commonplace. So until reentrant flex becomes commonplace, the solution was to include a pre-parsed psp_parser.c so that you woudln't need flex at all to compile mod_python. Looks like this still should be the case. The ./configure should just print a warning that if flex is not found or too old, should you need to rebuild psp_parser.c you will need to get the right version of flex. I've made the changes to configure.in, but before I commit I wanted to get some feedback. Are the following configure messages unclear or too verbose? For the case of the missing flex, ./configure will generate: ... checking for --with-flex... no checking for flex... no configure: WARNING: flex not found You can generally ignore this warning unless you need to regenerate psp_parser.c from psp_parse.l. If you do need regenerate psp_parser.c, use --with-flex to specify the location of flex. See the README for more information. ... For the case of the wrong flex version, ./configure will generate: ... checking for --with-flex... no checking for flex... /usr/local/sbin/flex found /usr/local/sbin/flex, we'll use this. Use --with-flex to specify another. checking flex version... configure: WARNING: Flex version 2.5.4 found. Version 2.5.31 or greater is required. You can generally ignore this warning unless you need to regenerate psp_parser.c from psp_parse.l. If you do need regenerate psp_parser.c, use --with-flex to specify the location of the correct flex version. See the README for more information. ... Any comments? Jim
Getting ready for 3.2 beta 2
I think we should aim for the second beta release in the next couple of days. I have a few questions and a list of outstanding issues. Name of tarball: mod_python-3.2.1b.tgz? Also, I assume a new branch called tags/release-3.2.1-BETA will be created in subversion, correct? Outstanding issues: 1. flex Fix is ready to commit pending some feedback on the warning message generated by configure. 2. MacOS compile Fixed in svn trunk. 3. Eliminate creation of mod_python_so.so in non-windows environments. Fix is ready to commit. 4. Fix segfault + memory leaks detailed in: http://issues.apache.org/jira/browse/MODPYTHON-75 http://issues.apache.org/jira/browse/MODPYTHON-60 Boyan's patch detailed in MODPYTHON-75 seems to fix both of these. Fix is ready to commit. 5. http://issues.apache.org/jira/browse/MODPYTHON-72 Fix still required. 6. Publisher bug in 3.2 BETA, detailed by Graham in python-dev message posted 2005-08-21. Fix still required. I haven't looked at the code involved in items 5 and 6, but hopefully the fixes will be fairly trivial. Regards, Jim
Re: Getting ready for 3.2 beta 2
Nicolas Lehuen wrote: Hi Jim, The fix for MODPYTHON-72 http://issues.apache.org/jira/browse/MODPYTHON-72 should be easy, unfortunately I'm quite busy right now, since my first daughter was born three days ago... Congratulations Nicolas! I'll do my best to have a look at it, but if someone feels like doing it, I'll understand. I have nothing planned for this weekend, so don't worry if you don't have the time. I should be able to handle it. Best Regards, Jim
Re: Publisher bug in 3.2 BETA.
I was hoping there would be a simple fix for this but a quick glance at the code makes me think that will not be the case. Also, I don't think this is a new bug as 3.1.4 does not generate a 404 NOT FOUND response either: Mod_python error: PythonHandler mod_python.publisher Traceback (most recent call last): File /usr/lib/python2.3/site-packages/mod_python/apache.py, line 299, in HandlerDispatch result = object(req) File /usr/lib/python2.3/site-packages/mod_python/publisher.py, line 98, in handler path=[path]) File /usr/lib/python2.3/site-packages/mod_python/apache.py, line 454, in import_module f, p, d = imp.find_module(parts[i], path) ImportError: No module named index We should create a new JIRA issue. If it turns out that the fix is not easy, is this a big enough problem to hold up the 3.2 release? Regards, Jim Graham Dumpleton wrote: In 3.2 beta, if one uses: SetHandler mod_python PythonHandler mod_python.publisher PythonDebug On and you DO NOT have an index.py file and use a URL where the first URL component doesn't map explicitly to a .py file, you will get a Python error rather than a 404 NOT FOUND response. For example, for URL http://localhost:8080/~grahamd/mp32/index;, I get: Mod_python error: PythonHandler mod_python.publisher Traceback (most recent call last): File /System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/mod_python/apache.py, line 299, in HandlerDispatch result = object(req) File /System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/mod_python/publisher.py, line 192, in handler module = page_cache[req] File /System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/mod_python/cache.py, line 77, in __getitem__ return self._checkitem(name)[2] File /System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/mod_python/cache.py, line 118, in _checkitem opened = self.check(key, name, entry) File /System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/mod_python/publisher.py, line 67, in check return ModuleCache.check(self, key, req, entry) File /System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/mod_python/cache.py, line 249, in check opened = file(key, self.mode) IOError: [Errno 2] No such file or directory: '/Users/grahamd/Sites/mp32/index.py'
Re: [jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Boyan,
How critical is this bug with respect to releasing 3.2? I'm heading out
for the weekend in the next couple of hours and was hoping to release
3.2.1beta beforehand. If this is a must fix I'll wait until next week.
(Either way, I'll fix the memory leak get_interpreter which you detail
in MODPYTHON-77).
Regards,
Jim
Jim Gallacher (JIRA) wrote:
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12320905 ]
Jim Gallacher commented on MODPYTHON-77:
I tried your patch and the unit tests fail. Apache fails to start, with the
following message in the error log:
Fatal Python error: PyThreadState_Get: no current thread
Using
GCC 4.0.2
apache mpm-prefork 2.0.54
python 2.3.5
Also, please submit your patches as unified diffs (diff -u) or even better if you are
using code checked out of subversion use svn diff
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3
--
Key: MODPYTHON-77
URL: http://issues.apache.org/jira/browse/MODPYTHON-77
Project: mod_python
Type: Bug
Components: core
Versions: 3.1.4
Environment: Python = 2.3
Reporter: Boyan Boyadjiev
Attachments: diff.txt, mod_python.c
The multiple interpreter concept of mod_python is broken for Python extension
modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter
Lock Acquisition for Extensions):
...
Limitations and Exclusions
This proposal identifies a solution for extension authors with
complex multi-threaded requirements, but that only require a
single PyInterpreterState. There is no attempt to cater for
extensions that require multiple interpreter states. At the time
of writing, no extension has been identified that requires
multiple PyInterpreterStates, and indeed it is not clear if that
facility works correctly in Python itself.
...
For mod_python this means, that complex Python extensions won't work any more with Python = 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
One solution, which works fine for me, is to save the first interpreter state into the interpreters dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
apr_pool_t *plog, server_rec *s)
{
...
/* initialize global Python interpreter if necessary */
if (! Py_IsInitialized())
{
/* initialze the interpreter */
Py_Initialize();
#ifdef WITH_THREAD
/* create and acquire the interpreter lock */
PyEval_InitThreads();
#endif
/* create the obCallBack dictionary */
interpreters = PyDict_New();
if (! interpreters) {
ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
python_init: PyDict_New() failed! No more memory?);
exit(1);
}
{
/*
Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition
for Extensions
BEGIN
*/
PyObject *p = 0;
interpreterdata * idata = (interpreterdata
*)malloc(sizeof(interpreterdata));
PyThreadState* currentThreadState = PyThreadState_Get();
PyInterpreterState *istate = currentThreadState-interp;
idata-istate = istate;
/* obcallback will be created on first use */
idata-obcallback = NULL;
p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p-refcout = 1*/
PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p);
/*p-refcout = 2*/
Py_DECREF(p); /*p-refcout = 1*/
/*
END
Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition
for Extensions
*/
}
/* Release the thread state because we will never use
* the main interpreter, only sub interpreters created later. */
PyThreadState_Swap(NULL);
#ifdef WITH_THREAD
/* release the lock; now other threads can run */
PyEval_ReleaseLock();
#endif
}
return OK;
}
Another change I've made in the attached file is to Py_DECREF(p) in
get_interpreter, which will remove leaky reference to the PyCObject with the
interpreter data. This was not a real problem, but now I see fewer leaks in
BoundsChecker :-).
Re: Getting ready for 3.2 beta 2
Gregory (Grisha) Trubetskoy wrote: I've been away this weekend - just got back, but I'm too busy to try to read all the multiple-interpreter related comments. I guess my question is - can someone provide a quick summary of how far we are from 3.2.1b test tarbal? I've also been away for the weekend. The patch for MODPYTHON-77 from last Thursday was causing apache to segfault and I have not had a chance to try the lastest changes from Boyan. As Graham stated on the weekend, the use of thread states can be very tricky. I think we should proceed with the 3.2.1b without the fix. That way we can take the time to make sure we understand the issues and fix it in 3.3. If that seems reasonable, I'll make the tarball today. Jim Thanks! Grisha On Thu, 1 Sep 2005, Graham Dumpleton wrote: Nicolas Lehuen wrote .. Well I for one am happy woth MODPYTHON-73, I've integrated Graham's patch and made unit test to check if everything was OK. Graham should be happy too :). As troublesome as I am, even I am happy at this point. :-) Unfortunately, probably will not be able to do any last build checks on MacOSX. I think I'll get killed tonight if I start working on the computer tonight since I fly off quite early tomorrow morning. If I am lucky I'll get just enough time to sync from the svn repository and then I'll play with it on the plane. I'll then be off the Internet for about 4-5 days so if there are any problems you will not hear about them in time anyway. Graham
mod_python 3.2.1b available for testing
A new mod_python 3.2 beta tarball is now available for testing. A Windows binary for python 2.4 is also provided. Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_ list*! (Not the [EMAIL PROTECTED] list, and preferably not me personally). The files are (temporarily) available here: http://www.modpython.org/dist/ Please download it, then do the usual $ ./configure --with-apxs=/wherever/it/is $ make $ (su) # make install Then (as non-root user!) $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the details (the versions of OS, Python and Apache, the test output, and suggestions, if any). Thank you, Jim Gallacher
Re: mod_python 3.2.1b available for testing
I agree with Nicolas. If we start chasing apache 2.1 alpha support we
might *never* get 3.2 out the door. :-( Also, what's the roadmap for
apache 2.1/2.2?
Jim
Nicolas Lehuen wrote:
Hi,
Could we focus on Apache 2.0 for the 3.2 release ? Put 2.1 on the
agenda for a later release (why not 3.3 ?).
For the moment I don't see any quick and easy way to support both 2.0
and 2.1, from what you wrote. I'd rather we try to get 3.2 out with a
proper 2.0 support, and try to fix things for 2.1 when it goes at
least beta.
Regards,
Nicolas.
2005/9/7, Jorey Bump [EMAIL PROTECTED]:
Jim Gallacher wrote:
A new mod_python 3.2 beta tarball is now available for testing. A
Windows binary for python 2.4 is also provided.
Please download it, then do the usual
$ ./configure --with-apxs=/wherever/it/is
$ make
$ (su)
# make install
Then (as non-root user!)
$ cd test
$ python test.py
And see if any tests fail. If they pass, send a +1 to the list, if they
fail, send the details (the versions of OS, Python and Apache, the test
output, and suggestions, if any).
-1
Slackware Linux 10.1
Python 2.4.1
Apache 2.1.6 Alpha
To fix:
Before running ./config, remove this from src/filterobject.c:
if (!APR_STATUS_IS_EAGAIN(self-rc) !APR_STATUS_IS_SUCCESS(self-rc)) {
PyErr_SetObject(PyExc_IOError,
PyString_FromString(Input filter read error));
return NULL;
}
And remove this from src/connobject.c:
if (! APR_STATUS_IS_SUCCESS(rc)) {
PyErr_SetObject(PyExc_IOError,
PyString_FromString(Connection read error));
return NULL;
}
Then configure/make/make install/test:
-1
Modify test.py for Apache 2.1.6-alpha, which has replaced mod_auth with
mod_auth_basic:
IfModule(!mod_auth_basic.c,
LoadModule(auth_basic_module %s %
quoteIfSpace(os.path.join(modpath, mod_auth_basic.so)
Run test.py again:
-1
Now all tests fail with socket error (111, 'Connection refused'). Since
the errors are consistent, this might be an easy fix, I just haven't
figured out the problem, yet.
Anyway, starting Apache normally loads mod_python fine, and I can run a
simple Hello, World! module using Publisher.
I know Apache 2.1.6-alpha is still a moving target, but hopefully this
will indicate some potential problems with the forthcoming Apache 2.2.x
stable branch that can be fixed now, such as the (deprecated?)
APR_STATUS_IS_SUCCESS references.
Re: mod_python 3.2.1b available for testing
+1 Linux Debian Sid apache 2.0.54 python 2.3.5 GCC 4.0.2
Re: mod_python 3.2.1b available for testing
Ron Reisor wrote: Yes! Plus, the software I'm developing is working too. I pulled out an early version of FileSession and started using Session.FileSession. Even better, you can use Session.Session() and the PythonOption session FileSession configuration directive to get all the benefits of FileSession without changing your current code - assuming of course that your current code is calling Session.Session() to create the session object. Jim Looks good. Gotta love it! thanks, Ron On Wed, 7 Sep 2005, Nicolas Lehuen wrote: Wow, that's a nice one :). 2005/9/7, Ron Reisor [EMAIL PROTECTED]: +1 MacOSX 10.4.2 gcc-4.0.0 (Apple build) Python 2.4.1 Apache 2.0.54 cheers, Ron Ron Reisor [EMAIL PROTECTED] (RWR3) University of Delaware Information Technologies/Network and Systems Services Computing Center/192 South Chapel Street/Newark DE, 19716 pgp finger print: 0D 73 06 6F D3 6A 99 D3 F5 D5 6E FF 3B B9 7C 2C Ron Reisor [EMAIL PROTECTED] (RWR3) University of Delaware Information Technologies/Network and Systems Services Computing Center/192 South Chapel Street/Newark DE, 19716 pgp finger print: 0D 73 06 6F D3 6A 99 D3 F5 D5 6E FF 3B B9 7C 2C
Re: FeeBSD build (was mod_python 3.2.1b available for testing)
Gregory (Grisha) Trubetskoy wrote: Anybody got FreeBSD? I'm getting this. This is an old and possibly misconfigured system, so the problem could be on my end. FreeBSD 4.9 apache 2.0.53 (from ports) python 2.3.3 $ make Compiling for DSO. /usr/local/sbin/apxs -I/home/grisha/src/tmp/mod_python-3.2.1b/src/include -I/usr/local/include/apache2 -I/usr/local/include/python2.3 -c mod_python.c _apachemodule.c requestobject.c tableobject.c util.c serverobject.c connobject.c filterobject.c hlist.c hlistobject.c -Wl,--export-dynamic -pthread -lm /usr/local/lib/python2.3/config/libpython2.3.a-lutil -lm /usr/local/share/apache2/build/libtool --silent --mode=compile cc -prefer-pic -O -pipe -DAP_HAVE_DESIGNATED_INITIALIZER -D_REENTRANT -D_THREAD_SAFE -I/usr/local/include/apache2 -I/usr/local/include/apache2 -I/usr/local/include/apache2 -I/usr/local/include -I/home/grisha/src/tmp/mod_python-3.2.1b/src/include -I/usr/local/include/apache2 -I/usr/local/include/python2.3 -c -o mod_python.lo mod_python.c touch mod_python.slo mod_python.c:34: syntax error before `*' mod_python.c:34: warning: data definition has no type or storage class mod_python.c: In function `python_cleanup': mod_python.c:238: warning: passing arg 1 of `free' discards qualifiers from pointer target type mod_python.c: In function `python_init': mod_python.c:517: `APR_THREAD_MUTEX_UNNESTED' undeclared (first use in this function) mod_python.c:517: (Each undeclared identifier is reported only once mod_python.c:517: for each function it appears in.) apxs:Error: Command failed with rc=65536 . *** Error code 1 Stop in /usr/home/grisha/src/tmp/mod_python-3.2.1b/src. *** Error code 1 Stop in /usr/home/grisha/src/tmp/mod_python-3.2.1b. I don't have FreeBSD, or any experience with any BSD, but I won't let that stop me from commenting. :) I don't see apr-0 listed in your includes in the above output. APR_THREAD_MUTEX_UNNESTED is defined in apr_thread_mutex.h, which on debian is in /usr/include/apr-0/. class ModPyExtension in dist/setup.py.in has been modified somewhat since the last release. This either broke the FreeBSD build or something is indeed misconfigured on your system. Jim
Re: FeeBSD build (was mod_python 3.2.1b available for testing)
Gregory (Grisha) Trubetskoy wrote: On Thu, 8 Sep 2005, Jim Gallacher wrote: I don't have FreeBSD, or any experience with any BSD, but I won't let that stop me from commenting. :) I don't see apr-0 listed in your includes in the above output. APR_THREAD_MUTEX_UNNESTED is defined in apr_thread_mutex.h, which on debian is in /usr/include/apr-0/. class ModPyExtension in dist/setup.py.in has been modified somewhat since the last release. This either broke the FreeBSD build or something is indeed misconfigured on your system. 'cept that ModPyExtension doesn't do anything on non-Win32... all those args come from apxs. Right you are. However, I still don't see any reference to include/apr-0/ in your output. For example, on debian the first few lines of the output from make: make[1]: Entering directory `/tmp/mod_python/src' Compiling for DSO. /usr/bin/apxs2 -I/tmp/mod_python/src/include -I/usr/include/apache2 -I/usr/include/python2.3 -c mod_python.c _apachemodule.c requestobject.c tableobject.c util.c serverobject.c connobject.c filterobject.c hlist.c hlistobject.c -L/usr/lib/python2.3/config -Xlinker -export-dynamic -lm -lpython2.3 -lpthread -ldl -lutil -lm /usr/bin/libtool --silent --mode=compile gcc -prefer-pic -pipe -I/usr/include/xmltok -I/usr/include/openssl -Wall -O2 -DAP_HAVE_DESIGNATED_INITIALIZER -DLINUX=2 -D_REENTRANT -D_XOPEN_SOURCE=500 -D_BSD_SOURCE -D_SVID_SOURCE -D_GNU_SOURCE -pipe -I/usr/include/xmltok -I/usr/include/openssl -Wall -O2 -pthread -I/usr/include/apache2 -I/usr/include/apr-0 -I/usr/include/apr-0 -I/usr/include -I/tmp/mod_python/src/include -I/usr/include/apache2 -I/usr/include/python2.3 -c -o mod_python.lo mod_python.c touch mod_python.slo I assume that apxs calls libtool, and provides the -I/usr/include/apr-0 parameter, yet I didn't see that in your output. I don't know if that is significant, just that it's different on the 2 systems. Jim
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
FYI, I found the following note in the INSTALL file in the apache source:
* If you are building on FreeBSD, be aware that threads will
be disabled and the prefork MPM will be used by default,
as threads do not work well with Apache on FreeBSD. If
you wish to try a threaded Apache on FreeBSD anyway, use
./configure --enable-threads.
I'm also setting up FreeBSD under QEMU... so far so good, but installing
anything using ports is really slow. QEMU's performance here is just
killing me. I guess I should have read the manual first and used the
binary packages for the software I wanted to install. :-(
Regards,
Jim
Jim Gallacher wrote:
Nicolas Lehuen wrote:
OK, I've checked in a version that compiles both on at least Win32 and
FreeBSD. I'm just testing if APR_HAS_THREAD is defined and only
include the apr_thread_mutex_lock and unlock calls if it is defined.
Compiles a passes unit tests on Linux Debian sid with mpm-prefork.
Now, on minotaur, APR_HAS_THREAD is defined as 0. Does this mean that
Apache is not configured for threading ? Can we assume that we are in
the prefork model if APR_HAS_THREAD==0, so that we can skip all the
locking code ? Because that's what we do right now.
On Debian sid with apache2.0.54 mpm-prefork, APR_HAS_THREAD == 1.
Jim
Regards,
Nicolas
2005/9/11, Nicolas Lehuen [EMAIL PROTECTED]
mailto:[EMAIL PROTECTED]:
Yes, this new code is something I commited on the 29/12/2004 (I used
the blame function of TortoiseSVN for that). It was a patch by
Graham to fix MODPYTHON-2
http://issues.apache.org/jira/browse/MODPYTHON-2.
The problem is not in the patch, but rather in the fact that APR
seems configured without the thread support while Python is
configured with thread support. mod_python.c assumes that is
WITH_THREAD is defined, then the APR mutex functions are available,
which is wrong. Maybe we should test for APR_HAS_THREADS instead ?
In that case, won't this cause any problems on threaded platforms ?
I don't know if this is a problem specific to minotaur or to all
version of FreeBSD. I'm currently downloading the ISOs of FreeBSD
and I'll try using QEMU to run a FreeBSD setup on my computer, but
that will be long and troublesome. If someone has more clue on this
issue, feel free to tell us :).
Regards,
Nicolas
2005/9/10, Jim Gallacher [EMAIL PROTECTED]
mailto:[EMAIL PROTECTED]:
I'm stubling around in the dark here, but maybe this will create a
spark
of an idea. I took a diff of mod_python.c from 3.1.4 and 3.2.1b and
isolated the lines which correspond to the compilation error.
Compiler messages
-
mod_python.c:34: error: syntax error before '*' token
mod_python.c:34: warning: type defaults to `int' in declaration of
`interpreters_lock'
mod_python.c:34: warning: data definition has no type or storage class
mod_python.c: In function `get_interpreter':
mod_python.c:131: warning: implicit declaration of function
`apr_thread_mutex_lock'
mod_python.c:161: warning: implicit declaration of function
`apr_thread_mutex_unlock'
mod_python.c: In function `python_init':
mod_python.c:517: warning: implicit declaration of function
`apr_thread_mutex_create'
mod_python.c:517: error: `APR_THREAD_MUTEX_UNNESTED' undeclared (first
use in this function)
Diff output
---
I've only copied the diff chunks which correspond to the complier
errors
mentioned above.
--- mod_python-3.1.4/src/mod_python.c Sat Jan 29 13:25:28 2005
+++ mod_python-3.2.1b/src/mod_python.c Tue Sep 6 17:11:03 2005
@@ -31,6 +31,8 @@
* (In a Python dictionary) */
static PyObject * interpreters = NULL;
+static apr_thread_mutex_t* interpreters_lock = 0;
+
apr_pool_t *child_init_pool = NULL;
... snip ...
@@ -124,11 +128,15 @@
name = MAIN_INTERPRETER;
#ifdef WITH_THREAD
+apr_thread_mutex_lock(interpreters_lock);
PyEval_AcquireLock();
#endif
... snip ...
@@ -149,6 +158,7 @@
#ifdef WITH_THREAD
PyEval_ReleaseLock();
+apr_thread_mutex_unlock(interpreters_lock);
#endif
... snip ...
@@ -490,13 +506,15 @@
}
/* initialize global Python interpreter if necessary */
-if (! Py_IsInitialized())
+if (initialized == 0 || !Py_IsInitialized())
{
-
+initialized = 1;
+
/* initialze the interpreter */
Py_Initialize();
#ifdef WITH_THREAD
+
apr_thread_mutex_create(interpreters_lock,APR_THREAD_MUTEX_UNNESTED,p);
/* create and acquire the interpreter lock */
PyEval_InitThreads();
#endif
So it would seem that the code causing the compile problems is new
for 3.2.
I also notice that in apr_arch_thread_mutex.h the typedef for
apr_thread_mutex_t is wrapped by #if APR_HAS_THREADS / #endif.
Looking at the apache source in srclib/apr/locks/unix/thread_mutex.c,
everything is also enclosed by #if APR_HAS_THREADS
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
Nicolas Lehuen wrote: OK, so on a non-threaded Apache, we can suppose we will be using the prefork MPM, so we don't need any code to support threading in mod_python, then, right ? Makes sense to me. In this case instead of testing for WITH_THREAD in mod_python.c : #ifdef WITH_THREAD maybe we could test for WITH_THREAD and APR_HAS_THREADS : #if APR_HAS_THREADS defined(WITH_THREAD) Right ? This would remove all threading-related code from mod_python when only prefork is available or when Python isn't compiled to support threads (I which case I wonder how it works in a threaded Apache...). Seems reasonable. It compiles and passes the unit tests on debian. I have given up using QEMU for now, minotaur is sufficient to make sure mod_python builds on FreeBSD. Granted, it won't allow me to give any +1 since I cannot run the unit tests (or can I ?)... I got FreeBSD running under QEMU, and the results are the same as on minotaur. With the #if APR_HAS_THREADS defined(WITH_THREAD) code change, mod_python compiles but the final link step generates a warning on both minotaur and qemu: /usr/local/share/apache2/build/libtool --silent --mode=link cc -o mod_python.la -rpath /usr/local/libexec/apache2 -module -avoid-version hlistobject.lo hlist.lo filterobject.lo connobject.lo serverobject.lo util.lo tableobject.lo requestobject.lo _apachemodule.lo mod_python.lo -Wl,--export-dynamic -pthread -lm /usr/local/lib/python2.4/config/libpython2.4.a -lutil -lm *** Warning: Linking the shared library mod_python.la against the *** static library /usr/local/lib/python2.4/config/libpython2.4.a is not portable! Likewise, the unit tests on both QEMU and minotaur fail. On minotaur: Syntax error on line 44 of /home/jgallacher/tmp/mod_python/test/conf/test.conf: Cannot load /home/jgallacher/tmp/mod_python/src/mod_python.so into server: /home/jgallacher/tmp/mod_python/src/mod_python.so: Undefined symbol btowc On qemu: Syntax error on line 44 of /usr/home/jim/tmp/mod_python/test/conf/test.conf: Cannot load /usr/home/jim/tmp/mod_python/src/mod_python.so into server: /usr/home/jim/tmp/mod_python/src/mod_python.so: Undefined symbol pthread_attr_init It is quite possible I don't have things configured correctly on the QEMU version and hence the different undefined symbol but it doesn't really matter since it fails either way. I don't have time to investigate further right now. I'll revisit this tonight. Regards, Jim Regards, Nicolas #if APR_HAS_THREADS defined(WITH_THREAD) 2005/9/11, Jim Gallacher [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: FYI, I found the following note in the INSTALL file in the apache source: * If you are building on FreeBSD, be aware that threads will be disabled and the prefork MPM will be used by default, as threads do not work well with Apache on FreeBSD. If you wish to try a threaded Apache on FreeBSD anyway, use ./configure --enable-threads. I'm also setting up FreeBSD under QEMU... so far so good, but installing anything using ports is really slow. QEMU's performance here is just killing me. I guess I should have read the manual first and used the binary packages for the software I wanted to install. :-( Regards, Jim Jim Gallacher wrote: Nicolas Lehuen wrote: OK, I've checked in a version that compiles both on at least Win32 and FreeBSD. I'm just testing if APR_HAS_THREAD is defined and only include the apr_thread_mutex_lock and unlock calls if it is defined. Compiles a passes unit tests on Linux Debian sid with mpm-prefork. Now, on minotaur, APR_HAS_THREAD is defined as 0. Does this mean that Apache is not configured for threading ? Can we assume that we are in the prefork model if APR_HAS_THREAD==0, so that we can skip all the locking code ? Because that's what we do right now. On Debian sid with apache2.0.54 mpm-prefork, APR_HAS_THREAD == 1. Jim Regards, Nicolas 2005/9/11, Nicolas Lehuen [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] mailto: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: Yes, this new code is something I commited on the 29/12/2004 (I used the blame function of TortoiseSVN for that). It was a patch by Graham to fix MODPYTHON-2 http://issues.apache.org/jira/browse/MODPYTHON-2. The problem is not in the patch, but rather in the fact that APR seems configured without the thread support while Python is configured with thread support. mod_python.c assumes that is WITH_THREAD is defined, then the APR mutex functions are available, which is wrong. Maybe we should test for APR_HAS_THREADS instead ? In that case, won't this cause any problems on threaded platforms
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
Gregory (Grisha) Trubetskoy wrote: Shouldn't that be PYTHON_WITH_THREAD rather than MOD_PYTHON_WITH_THREAD? I understand it to mean that we want the thread handling code compiled into mod_python. Compiling and testing right now. Jim On Mon, 12 Sep 2005, Nicolas Lehuen wrote: I've checked in a changeset wherein I define MOD_PYTHON_WITH_THREAD_SUPPORT and use it everywhere WITH_THREAD was previously used. This should do the trick ! Now if someone (like Jim) can give us his +1, that would be great. Regards, Nicolas 2005/9/12, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: Just wanted to add to this message that if Jim's version runs and tests with the trick below (envvars is executed prior to apache start, but I don't think the tests use it, so you'll probably just have to set this var in the shell in which the tests are run), then this would be a solution for all FreeBSD issues and we could roll a beta 3 which will have a great change of being publicly released. Grisha On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: OK, found it. This should work on FreeBSD where Python is threaded and Apache is not. [snip] And, if you built apache without thread support, you may need to add the following lines to $PREFIX/sbin/envvars: LD_PRELOAD=/usr/lib/libc_r.so export LD_PRELOAD [snip] On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: On Mon, 12 Sep 2005, Jim Gallacher wrote: *** Warning: Linking the shared library mod_python.la against the *** static library /usr/local/lib/python2.4/config/libpython2.4.a is not portable! I think this was always there and its pretty harmless. On qemu: Syntax error on line 44 of /usr/home/jim/tmp/mod_python/test/conf/test.conf: Cannot load /usr/home/jim/tmp/mod_python/src/mod_python.so into server: /usr/home/jim/tmp/mod_python/src/mod_python.so: Undefined symbol pthread_attr_init This is because FreeBSD's libc comes in two versions - threaded and non-threaded. If Python is linked against the threaded ones and Apache against the non-thrreaded, then you get this problem. There is a simple fix for this - you just cause Apache to start with threaded libs, but I can't find any references to it right now and have to run off to a meeting. Grisha It is quite possible I don't have things configured correctly on the QEMU version and hence the different undefined symbol but it doesn't really matter since it fails either way. I don't have time to investigate further right now. I'll revisit this tonight. Regards, Jim Regards, Nicolas #if APR_HAS_THREADS defined(WITH_THREAD) 2005/9/11, Jim Gallacher [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: FYI, I found the following note in the INSTALL file in the apache source: * If you are building on FreeBSD, be aware that threads will be disabled and the prefork MPM will be used by default, as threads do not work well with Apache on FreeBSD. If you wish to try a threaded Apache on FreeBSD anyway, use ./configure --enable-threads. I'm also setting up FreeBSD under QEMU... so far so good, but installing anything using ports is really slow. QEMU's performance here is just killing me. I guess I should have read the manual first and used the binary packages for the software I wanted to install. :-( Regards, Jim Jim Gallacher wrote: Nicolas Lehuen wrote: OK, I've checked in a version that compiles both on at least Win32 and FreeBSD. I'm just testing if APR_HAS_THREAD is defined and only include the apr_thread_mutex_lock and unlock calls if it is defined. Compiles a passes unit tests on Linux Debian sid with mpm-prefork. Now, on minotaur, APR_HAS_THREAD is defined as 0. Does this mean that Apache is not configured for threading ? Can we assume that we are in the prefork model if APR_HAS_THREAD==0, so that we can skip all the locking code ? Because that's what we do right now. On Debian sid with apache2.0.54 mpm-prefork, APR_HAS_THREAD == 1. Jim Regards, Nicolas 2005/9/11, Nicolas Lehuen [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] mailto: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: Yes, this new code is something I commited on the 29/12/2004 (I used the blame function of TortoiseSVN for that). It was a patch by Graham to fix MODPYTHON-2 http://issues.apache.org/jira/browse/MODPYTHON-2. The problem is not in the patch, but rather in the fact that APR seems configured without the thread support while Python is configured with thread support. mod_python.c assumes that is WITH_THREAD is defined, then the APR mutex functions are available, which is wrong. Maybe we should test for APR_HAS_THREADS instead ? In that case, won't this cause any problems on threaded platforms ? I don't know if this is a problem specific to minotaur or to all version of FreeBSD. I'm currently downloading the ISOs of FreeBSD and I'll try using QEMU to run a FreeBSD setup on my computer
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
we're doing here so there is a record. Sorry if I'm being slow-headed here. To me it seems that when you use thread-related calls from Python, you wrap those in Python defines (WITH_THREAD) and when you use thread-related calls from APR, you wrap those in APR defines (APR_HAS_THREAD), and that's all? In other words - what does MOD_PYTHON_WITH_THREAD_SUPPORT accomplish that the above does not. Also, given: #if(defined(WITH_THREAD) APR_HAS_THREADS) #define MOD_PYTHON_WITH_THREAD_SUPPORT 1 #else #define MOD_PYTHON_WITH_THREAD_SUPPORT 0 #endif Does this mean that if Python is compiled with thread support and APR is not, MOD_PYTHON_WITH_THREAD_SUPPORT is 0 which means that the thread safety code isn't there, but you still _can_ create threads because Python will let you - isn't this asking for a segfault/deadlock/whatever? Grisha On Mon, 12 Sep 2005, Jim Gallacher wrote: Gregory (Grisha) Trubetskoy wrote: Shouldn't that be PYTHON_WITH_THREAD rather than MOD_PYTHON_WITH_THREAD? I understand it to mean that we want the thread handling code compiled into mod_python. Compiling and testing right now. Jim On Mon, 12 Sep 2005, Nicolas Lehuen wrote: I've checked in a changeset wherein I define MOD_PYTHON_WITH_THREAD_SUPPORT and use it everywhere WITH_THREAD was previously used. This should do the trick ! Now if someone (like Jim) can give us his +1, that would be great. Regards, Nicolas 2005/9/12, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]: Just wanted to add to this message that if Jim's version runs and tests with the trick below (envvars is executed prior to apache start, but I don't think the tests use it, so you'll probably just have to set this var in the shell in which the tests are run), then this would be a solution for all FreeBSD issues and we could roll a beta 3 which will have a great change of being publicly released. Grisha On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: OK, found it. This should work on FreeBSD where Python is threaded and Apache is not. [snip] And, if you built apache without thread support, you may need to add the following lines to $PREFIX/sbin/envvars: LD_PRELOAD=/usr/lib/libc_r.so export LD_PRELOAD [snip] On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote: On Mon, 12 Sep 2005, Jim Gallacher wrote: *** Warning: Linking the shared library mod_python.la against the *** static library /usr/local/lib/python2.4/config/libpython2.4.a is not portable! I think this was always there and its pretty harmless. On qemu: Syntax error on line 44 of /usr/home/jim/tmp/mod_python/test/conf/test.conf: Cannot load /usr/home/jim/tmp/mod_python/src/mod_python.so into server: /usr/home/jim/tmp/mod_python/src/mod_python.so: Undefined symbol pthread_attr_init This is because FreeBSD's libc comes in two versions - threaded and non-threaded. If Python is linked against the threaded ones and Apache against the non-thrreaded, then you get this problem. There is a simple fix for this - you just cause Apache to start with threaded libs, but I can't find any references to it right now and have to run off to a meeting. Grisha It is quite possible I don't have things configured correctly on the QEMU version and hence the different undefined symbol but it doesn't really matter since it fails either way. I don't have time to investigate further right now. I'll revisit this tonight. Regards, Jim Regards, Nicolas #if APR_HAS_THREADS defined(WITH_THREAD) 2005/9/11, Jim Gallacher [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: FYI, I found the following note in the INSTALL file in the apache source: * If you are building on FreeBSD, be aware that threads will be disabled and the prefork MPM will be used by default, as threads do not work well with Apache on FreeBSD. If you wish to try a threaded Apache on FreeBSD anyway, use ./configure --enable-threads. I'm also setting up FreeBSD under QEMU... so far so good, but installing anything using ports is really slow. QEMU's performance here is just killing me. I guess I should have read the manual first and used the binary packages for the software I wanted to install. :-( Regards, Jim Jim Gallacher wrote: Nicolas Lehuen wrote: OK, I've checked in a version that compiles both on at least Win32 and FreeBSD. I'm just testing if APR_HAS_THREAD is defined and only include the apr_thread_mutex_lock and unlock calls if it is defined. Compiles a passes unit tests on Linux Debian sid with mpm-prefork. Now, on minotaur, APR_HAS_THREAD is defined as 0. Does this mean that Apache is not configured for threading ? Can we assume that we are in the prefork model if APR_HAS_THREAD==0, so that we can skip all the locking code ? Because that's what we do right now. On Debian sid with apache2.0.54 mpm-prefork, APR_HAS_THREAD == 1. Jim Regards, Nicolas 2005/9/11, Nicolas Lehuen [EMAIL PROTECTED] mailto:[EMAIL
Re: FreeBSD compile problem (was Getting ready for 3.2 beta 2)
Gregory (Grisha) Trubetskoy wrote:
Done. All tests pass on my FreeBSD box. Nicolas - can you test Win32,
I'm not 100% sure if the change to test.py I made will work.
Good news. If the changes can be checked in and Nicolas can give a +1 on
the Windows test then I'll be able to generate the next, and hopefully
last, beta tonight.
As an aside, it may be useful to have an option for test.py to preserve
a copy of the apache logs. It would make troubleshooting the unit test
failures much easier. Currently test.py deletes the logs after each unit
test.
Regards,
Jim
Grisha
On Tue, 13 Sep 2005, Nicolas Lehuen wrote:
Yes, now I remember I had to comment this line out because it broke
something on Win32. Go ahead, uncomment it and I'll add a test to
remove it
when running the test on Win32.
Regards,
Nicolas
2005/9/13, Gregory (Grisha) Trubetskoy [EMAIL PROTECTED]:
OK, I found the problem. The LockFile line is commented out in test.py
which causes Apache to try to create the lock in the default location in
/var/run.
http://svn.apache.org/viewcvs.cgi/httpd/mod_python/trunk/test/test.py?rev=125771r1=106619r2=125771
So the question is - can we just put it back in, or does it break
something on Win32?
Cheers
Grisha
On Tue, 13 Sep 2005, Jim Gallacher wrote:
Nicolas Lehuen wrote:
Jim, do you manage to build and test the 3.1.4 version on your setup ?
This looks like a permission problem, not something related to our
current
problem.
I haven't tried 3.1.4. And I could also try the tests as root, which
would
eliminate any permission problems. I have a busy day ahead of me so
this
will
have to wait until tonight.
Jim
Regards,
Nicolas
2005/9/13, Jim Gallacher [EMAIL PROTECTED]
mailto:[EMAIL PROTECTED]:
-1 for this patch. Actually, the patch itself is fine - it just
doesn't
fix the problem. The unit tests are still failings as per my previous
messages. ie the following is getting logged in test/logs/error_log:
[Mon Sep 12 19:49:33 2005] [emerg] (2)No such file or directory:
Couldn't create accept lock
Jim
Gregory (Grisha) Trubetskoy wrote:
Here's a patch (this is against 3.1.2b). Untested. This replaces
GIL
with with the APR lock.
--- src/mod_python.c.orig Mon Sep 12 16:42:28 2005
+++ src/mod_python.c Mon Sep 12 17:32:26 2005
@@ -31,7 +31,9 @@
* (In a Python dictionary) */
static PyObject * interpreters = NULL;
+#ifdef APR_HAS_THREAD
static apr_thread_mutex_t* interpreters_lock = 0;
+#endif
apr_pool_t *child_init_pool = NULL;
@@ -127,9 +129,8 @@
if (! name)
name = MAIN_INTERPRETER;
-#ifdef WITH_THREAD
+#ifdef APR_HAS_THREAD
apr_thread_mutex_lock(interpreters_lock);
- PyEval_AcquireLock();
#endif
if (!interpreters) {
@@ -156,8 +157,7 @@
idata = (interpreterdata *)PyCObject_AsVoidPtr(p);
}
-#ifdef WITH_THREAD
- PyEval_ReleaseLock();
+#ifdef APR_HAS_THREAD
apr_thread_mutex_unlock(interpreters_lock);
#endif
@@ -513,8 +513,10 @@
/* initialze the interpreter */
Py_Initialize();
-#ifdef WITH_THREAD
+#ifdef APR_HAS_THREAD
apr_thread_mutex_create(interpreters_lock,APR_THREAD_MUTEX_UNNESTED,p);
+#endif
+#ifdef WITH_THREAD
/* create and acquire the interpreter lock */
PyEval_InitThreads();
#endif
On Mon, 12 Sep 2005, Gregory (Grisha) Trubetskoy wrote:
Yep, this is getting a little hairy, but nothing we couldn't
handle :-)
I did a little more research. Basically, this started with
Graham's
patch that addressed a problem with modules being reimported (or
something). From Graham's message:
The basic problem revolves around the Python dictionary used to
hold
the set of interpreters. The code in mod_python.c is trying to
use
the Python GIL to provide exclusive access to that dictionary
and any
subsequent creation of an interpreter.
The only catch is that in creating a new interpreter, the
Python core
is, in someway I don't understand, swapping thread states at some
point which is allowing other threads to acquire the GIL.
So what Graham's patch does is create an APR lock
(interpreters_lock)
and wrap all the access to the dictionary with calls to
apr_mutex_lock/unlock.
I think the _real_ way to address this issue is to first find
what is
the problem with using the Python GIL to serialize access to the
interpreters dictionary. Is this a Python bug, or are we not
understanding GIL and using it improperly?
BUT, given that the above question may be complicated to answer,
and
that Graham's patch resolves the issue, another thought:
If the APR lock works, what is the point of using the GIL in
addition?
Should we just use the APR-based lock alone? I.e., where we had
(after
Graham's patch):
#ifdef WITH_THREAD
apr_thread_mutex_lock(interpreters_lock);
PyEval_AcquireLock();
#endif
we would use:
#ifdef APR_HAS_THREAD
apr_thread_mutex_lock(interpreters_lock);
#endif
_without_ a call to PyEval_AcquireLock() at all.
It should compile OK, and on platforms where APR has no thread
support, like you said, it's
mod_python 3.2.2b available for testing
A new mod_python 3.2.2 beta tarball is now available for testing. Hopefully this will be the last beta before the official 3.2 release. Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_ list*! (Not the [EMAIL PROTECTED] list, and preferably not me personally). The files are (temporarily) available here: http://www.modpython.org/dist/ Please download it, then do the usual $ ./configure --with-apxs=/wherever/it/is $ make $ (su) # make install Then (as non-root user!) $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the details (the versions of OS, Python and Apache, the test output, and suggestions, if any). Thank you, Jim Gallacher
Re: mod_python 3.2.2b available for testing
+1 Linux Debian Sid apache 2.0.54 python 2.3.5 gcc 4.0.2 Jim Gallacher wrote: A new mod_python 3.2.2 beta tarball is now available for testing. Hopefully this will be the last beta before the official 3.2 release. Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_ list*! (Not the [EMAIL PROTECTED] list, and preferably not me personally). The files are (temporarily) available here: http://www.modpython.org/dist/ Please download it, then do the usual $ ./configure --with-apxs=/wherever/it/is $ make $ (su) # make install Then (as non-root user!) $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the details (the versions of OS, Python and Apache, the test output, and suggestions, if any). Thank you, Jim Gallacher
Re: mod_python 3.2.2b available for testing
Grisha, I think you mentioned that we should announce this beta on the mod_python list as well. If so I thought we could wait until we get a +1 from a MacOS X user here on python-dev before proceeding. Regards, Jim Jim Gallacher wrote: A new mod_python 3.2.2 beta tarball is now available for testing. Hopefully this will be the last beta before the official 3.2 release. Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_ list*! (Not the [EMAIL PROTECTED] list, and preferably not me personally). The files are (temporarily) available here: http://www.modpython.org/dist/ Please download it, then do the usual $ ./configure --with-apxs=/wherever/it/is $ make $ (su) # make install Then (as non-root user!) $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the details (the versions of OS, Python and Apache, the test output, and suggestions, if any). Thank you, Jim Gallacher
Re: mod_python 3.2.2b available for testing
Nicolas has created a windows binary for testing which is also available at http://www.modpython.org/dist. Regards, Jim Jim Gallacher wrote: A new mod_python 3.2.2 beta tarball is now available for testing. Hopefully this will be the last beta before the official 3.2 release. Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_ list*! (Not the [EMAIL PROTECTED] list, and preferably not me personally). The files are (temporarily) available here: http://www.modpython.org/dist/ Please download it, then do the usual $ ./configure --with-apxs=/wherever/it/is $ make $ (su) # make install Then (as non-root user!) $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the details (the versions of OS, Python and Apache, the test output, and suggestions, if any). Thank you, Jim Gallacher
Re: svn commit: r290569 - /httpd/mod_python/trunk/lib/python/mod_python/SQLiteSession.py
Gregory (Grisha) Trubetskoy wrote: OK, my next question would be - is MySQL, PostgreSQL, Informix, Oracle, etc next, Yes. ;) and is this the path we want to take, or is there something about sqlite that makes it unique? I don't know if it is that path we *want* to take, but I think there is a certain inevitability that it's the path we *will* take. I'm personally thinking about a MySQL backend, not because I need it but because I'm curious to compare the performance with FileSession. The best approach might be to refactor Session.py to easily accommodate subclasses for other persistent stores. Any new session subclasses get their own file eg. mod_python/sessions/sqlite.py mod_python/sessions/mysql.py That way there are no dependency problems in Session.py. If some code imports sessions.sqlite and sqlite is not installed then let it raise an exception as it should. The current Session.py could stay as is, or we could split FileSession, DbmSession and MemorySession into their own files and just have a stub in Session.py for backward compatability. This reorganization may also make it easier for users to create their own session subclasses.Related to this, the current code for Session.Session only allows one of the standard session classes to be specified by PythonOption session. It would be nice if it there was a way to make this more dynamic and the refactoring might assist in this. On another note, if we are starting to roll with new features for 3.3 I would suggest we need to immediately create a new svn branch for 3.2 bugfixes. Regards, Jim On Thu, 22 Sep 2005, Robert Sanderson wrote: Can we have a little discussion on pros/cons of this? Does this make mod_python dependent on sqlite? Nope. It'll just silently fail if it can't import dbapi2. Rob +try: +# If this code is included into Session.py, +# we don't want to add a dependency to SQLite +from pysqlite2 import dbapi2 as sqlite +except: +pass +else: +class SQLiteSession(BaseSession):
Re: svn commit: r290569 - /httpd/mod_python/trunk/lib/python/mod_python/SQLiteSession.py
Grisha, Your message implies that there is a mailing list for mod_python svn commit messages. How can I subscribe to this? Thanks, Jim Gregory (Grisha) Trubetskoy wrote: Can we have a little discussion on pros/cons of this? Does this make mod_python dependent on sqlite? Thanks, Grisha On Tue, 20 Sep 2005 [EMAIL PROTECTED] wrote: Author: nlehuen Date: Tue Sep 20 14:28:32 2005 New Revision: 290569 URL: http://svn.apache.org/viewcvs?rev=290569view=rev Log: A first try at implementing a session storage relying on SQLite. It is slower than FileSession but could scale better ? Added: httpd/mod_python/trunk/lib/python/mod_python/SQLiteSession.py Added: httpd/mod_python/trunk/lib/python/mod_python/SQLiteSession.py URL: http://svn.apache.org/viewcvs/httpd/mod_python/trunk/lib/python/mod_python/SQLiteSession.py?rev=290569view=auto
Re: SQLite and other things [was Re: svn commit: r290569]
Gregory (Grisha) Trubetskoy wrote: Just to put this SQLite business to rest. I think that (and we can discuss this - I don't set laws, I just have opinions that may not always beright, so feel free to comment) mod_python should do fewer things but do them exceptionally well. Roughly speaking, to be included in mod_python something has to fall into the category of Apache/Python integration or demonstrate to be addressing a challenge introduced by the mod_python environment and too difficult to address by an average (Python) programmer. Given this definition, the reason Sessions were included was that it turned out that implementing session support under mod_python was not a trivial task and required use of APR locking (which was implemented for Sessions IIRC). Similar reasoning behind the inclusion of PSP - it's not something an average Python programmer could whip up, but it is also a bit of a stretch and its inclusion was (and is) controversial. Cookies were included because the standard Python lib module assumed a CGI environment and was (and is) IMHO lacking in many ways. On a more practical note - interoperation with third-party applications creates a dependency. So even though SQLite would not have to be installed in order for mod_python to work, it _would_ have to be installed in order for the test suite to run. Or you don't test it, in which case you risk it becoming broken and not noticing it. And if we were to continue down this path, imagine having to have MySQL, PostgreSQL and Oracle installed in order to run the test suite - we'd never get any +1's on the list this way :-) So based on the above, I think SQLite support should be removed. The thing we need to address is what to do with nifty things we create but that don't qualify for inclusion. The idea of a 'contrib' directory has been floated around for a while, I for one am against it for the same reasons above - it should either be 100% supported or not included at all IMO. I think for the time being the best approach is for people to use their own resources to publish contrib-type code on the web and just follow the usual process of announcing it on the list. Grisha Grisha, I think much what you write could be distilled into The mod_python Philosophy and should included in the documenation as such. I'm ambivalent on the subject of a contrib directory. In theory it's a great idea. In practice it often seems that a project's contributed code is out of date and unmaintained. I think we should *only* ship production quality code. This means that we would need to take responsibilty for maintaining contributed code which I think would be a bad idea - our resources are too limited. It may be worthwhile expanding on our example code however. I don't think people expect examples to be full fledged implementations, but can be a great starting point for a user to create their own code. I guess we can still get into dependency problems here as well though. There are no easy answers. Regards, Jim
Re: testhandler.py [was SQLite and other things ]
Nick wrote: Jim Gallacher wrote: Nicolas Lehuen wrote: I thought that all this mptest.py thing was a bit disturbing, as usually people took the wrong impression that they had to call the /test/mptest.py URL, that is they thought that the handler system was a bit like the publisher system. So by providing a test handler in the box, we could make sure that the installation was correct without having to cope with all the fuss about where mptest.py should be located and what URL to use to make sure everything is working. I think this is a worthwhile addition as it should make it easier to get new people started with mod_python. Looking at this file, it's a great way to verify that mod_python is working correctly, and it's pretty thorough in checking the apache internals. However, it's neither simple nor intuitive for new people to get started with mod_python I think. For a newbie, it looks like it could be overwhelming. Allow me to rephrase. It is a worth while addition as it should make it easier *for experienced users* to assist newbies having problems getting started with mod_python. :) Regards, Jim
Re: Minor documentation error in 3.2.2b.
Graham Dumpleton wrote: Documentation says: path_info String. What follows after the file name, but is before query args, if anything. Same as CGI PATH_INFO. (Read-Only) The path_info member is now actually modifiable in 3.2 and so Read Only designator can be dropped. I've already fixed this, and as well as a couple of other minor documentation issues, but have not commited yet. I'll do that this weekend. BTW, what is the timetable for actual release of 3.2? Where I am working is finally upgrading to Apache 2.0 on one of their servers and want to be able to use final 3.2 release on the system. Grisha proposed the following release schedule on August 8. Gregory (Grisha) Trubetskoy wrote: Aug 153.2.0b private release Sep 73.2.Xbapproved for public release Nov 73.2.Yprivate release Nov 153.2.Ypublic release [I'm using X and Y since we don't know what those numbers may be] The Sep 7 - Nov 7 is a month for public testing of 3.2.Xb. The public 3.2.2b release annoucement was made on Sept 19, so we are almost on schedule. Jim
Re: glue between apache and python logging
Nicolas Lehuen wrote:
2005/10/19, Nic Ferrier [EMAIL PROTECTED]
mailto:[EMAIL PROTECTED]:
Is everyone here called Nic[h]olas?
Nicolas Lehuen [EMAIL PROTECTED]
mailto:[EMAIL PROTECTED] writes:
Nic, there is something I need to understand before giving my
advice on the
subject. I'm not familiar with the logging API, can you tell me
how you
configure Python to use this logging implementation ? Looks like
we have to
manipulate configuration object, set up handlers and so on... If
so I guess
we'd have to add a word or two in the documentation on the best
way to do
this in the mod_python context...
Hmmm... logging is actually very simple if you want it to be.
All you really have to do to use logging from Python is this:
import logging
# then sometime later
logger = logging.getLogger()
logger.debug(a log message)
lot's more things are possible and the logging API is very
configurable.
From an Apache point of view you do need to configure the default
logger in your handler so that it has access to the apache vhost
logger:
hdlr = apache_log_handler.ApacheLogHandler(http)
logging.getLogger().addHandler(hdlr)
once you've done that then any handler which descends from the default
handler (and that's the normal way to do things) will log to Apache.
Nic
In that case, setting up the logging handler should be done by the user,
making sure that it is set up only once per interpreter, even in the
context of a multi-threaded MPM. It's not a trivial thing ; looks like
this is a job for PythonImport.
Except that you won't have a server reference to get the virtual host
configuration. If you are using a custom log for each virtual host,
won't your error messages end up in the wrong log?
So my advice is that if the set up is done by the user, then the logging
handler should be considered as contrib. If you provide a way for
mod_python to directly set up the handler, maybe this could be
considered part of mod_python.
Here are some further things to consider if anyone wants to persue it.
Consider the following code:
import logging
from mod_python import apache
from proposed_mp_logging_module import ApacheLogHandler
def handler(req)
req.content_type = 'text/plain'
log = logging.getLogger('request')
hdlr = ApacheLogHandler(req)
log.addHandler(hdlr)
log.setLevel(logging.INFO)
msg = 'Its all good'
log.info('%s' % msg)
req.write('check the logs for %s' % msg)
apache.OK
All fine and dandy. But isn't logger a singleton? So each time a request
is processed we'll be adding a reference to the request object, which
will never get garbage collected and result in a memory leak.
Furthermore, you can't depend on the request object being valid once
the request processing has completed. At some point request_tp_clear (in
requestobject.c) will get called and request-server will be set to
NULL. (At least I think this is correct). My gut tells me you'll get
some random seg faults.
Also, is there not an assumption that the logger instance is in effect
global? So let's say you change the level in one request such as
log.setLevel(logging.CRITICAL). In mpm-prefork or mpm-worker models
changing the log level will not propagate to the other child processes.
I expect most users will find this confusing.
As I said before I don't think this is as trivial to implement as it
might seem at first blush.
Regards.
Jim
Re: Where do we stand on 3.2.2 final?
Nicolas Lehuen wrote: OK I've checked in Graham's fix for MODPYTHON-83, and unsurprisingly it doesn't break anything on Win32. So I'm ready to provide the next beta build this week end if we can. As for MODPYTHON-84 (the one about req.sendfile)I cannot really test it since there are no symbolic links in Win2k (only junctions and reparse points but that's another story). Likewise, I cannot add a unit test for it in the test suite since, well, I cannot test the unit test... Regards, Nicolas Sendfile calls apr_stat, which is OS specific. In req_sendfile apr_stat is now called with APR_FINFO_NORM instead of APR_READ. So although Windows may not have symbolic links I wanted to ensure the change didn't break normal req_sendfile behaviour as well. Having said that I realize this would be caught by the normal unit tests... doh! Regards, Jim
Re: Where do we stand on 3.2.2 final?
Nicolas Lehuen wrote: OK I've checked in Graham's fix for MODPYTHON-83, and unsurprisingly it doesn't break anything on Win32. So I'm ready to provide the next beta build this week end if we can. +1 I don't see any outstanding issues so if we are doing another beta let's do it today. Just a note on updating src/include/mpversion.h. Should we be changing MPV_PATCH in svn trunk (and by extension MPV_STRING) to track the beta release patch level? In trunk we are still at MPV_PATCH 0, but in tags/release-3-2-2b we are at MPV_PATCH 2. Regards, Jim
Re: Where do we stand on 3.2.2 final?
Nicolas Lehuen wrote: OK, I've added a unit test for req.send_file(fname) and everything seems OK under Windows. Now if anyone want to write a test for the symbolic link issue... Will do. Jim Regards, Nicolas 2005/10/22, Nicolas Lehuen [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: 2005/10/22, Jim Gallacher [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: Nicolas Lehuen wrote: OK I've checked in Graham's fix for MODPYTHON-83, and unsurprisingly it doesn't break anything on Win32. So I'm ready to provide the next beta build this week end if we can. As for MODPYTHON-84 (the one about req.sendfile)I cannot really test it since there are no symbolic links in Win2k (only junctions and reparse points but that's another story). Likewise, I cannot add a unit test for it in the test suite since, well, I cannot test the unit test... Regards, Nicolas Sendfile calls apr_stat, which is OS specific. In req_sendfile apr_stat is now called with APR_FINFO_NORM instead of APR_READ. So although Windows may not have symbolic links I wanted to ensure the change didn't break normal req_sendfile behaviour as well. Having said that I realize this would be caught by the normal unit tests... doh! Regards, Jim Well, no, because the current unit test for req.send_file provides values for offset and length. I'll add another test without offset and length, and check for the length of the result on the client side. Regards, Nicolas
mod_python 3.2.3b available for testing
A new mod_python 3.2.3 beta tarball is now available for testing. A Windows binary is also available. This release is similar to 3.2.2b but fixes a couple a small issues where a non-threaded python is used. Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_ list*! (Not the [EMAIL PROTECTED] list, and preferably not me personally). The files are (temporarily) available here: http://www.modpython.org/dist/ Please download it, then do the usual $ ./configure --with-apxs=/wherever/it/is $ make $ (su) # make install Then (as non-root user!) $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the details (the versions of OS, Python and Apache, the test output, and suggestions, if any). Thank you, Jim Gallacher
Re: mod_python 3.2.3b available for testing
Indrek Järve wrote:
Jim Gallacher wrote:
And see if any tests fail. If they pass, send a +1 to the list, if they
fail, send the details (the versions of OS, Python and Apache, the test
output, and suggestions, if any).
Thank you,
Jim Gallacher
+1 on SuSE Linux 9.2 (i586)
+1 on SuSE Linux 9.2 (x86-64) once I edited the configure script and
replaced lib/ with lib64/ in PY_STD_LIB and PyLIBP
I wonder how we might correctly determine the 'lib' part of the path.
The relevant section of configure.in is:
# find out compiled in install prefix
AC_MSG_CHECKING(Python install prefix)
PyEXEC_INSTALLDIR=`$PYTHON_BIN -c import sys; print sys.exec_prefix`
AC_MSG_RESULT($PyEXEC_INSTALLDIR)
# this is where the Python libraries will get installed
AC_SUBST(PY_STD_LIB)
PY_STD_LIB=${PyEXEC_INSTALLDIR}/lib/python${PyVERSION}
# set python std library variable
AC_SUBST(LIBS)
PyLIBP=${PyEXEC_INSTALLDIR}/lib/python${PyVERSION}
For those of you not familiar with autoconf, we use configure.in as the
source file to generate the configure file.
Regards,
Jim
Re: mod_python 3.2.3b available for testing
Indrek Järve wrote:
Graham Dumpleton wrote:
Jim Gallacher wrote ..
Indrek Järve wrote:
Jim Gallacher wrote:
And see if any tests fail. If they pass, send a +1 to the list, if
they
fail, send the details (the versions of OS, Python and Apache, the
test
output, and suggestions, if any).
Thank you,
Jim Gallacher
+1 on SuSE Linux 9.2 (i586)
+1 on SuSE Linux 9.2 (x86-64) once I edited the configure script and
replaced lib/ with lib64/ in PY_STD_LIB and PyLIBP
I wonder how we might correctly determine the 'lib' part of the path.
The relevant section of configure.in is:
# find out compiled in install prefix
AC_MSG_CHECKING(Python install prefix)
PyEXEC_INSTALLDIR=`$PYTHON_BIN -c import sys; print sys.exec_prefix`
AC_MSG_RESULT($PyEXEC_INSTALLDIR)
# this is where the Python libraries will get installed
AC_SUBST(PY_STD_LIB)
PY_STD_LIB=${PyEXEC_INSTALLDIR}/lib/python${PyVERSION}
# set python std library variable
AC_SUBST(LIBS)
PyLIBP=${PyEXEC_INSTALLDIR}/lib/python${PyVERSION}
For those of you not familiar with autoconf, we use configure.in as the
source file to generate the configure file.
Personally, I would try and use distutils to make these judgements
as to
where things live rather than constructing the standard lib directory
path
from prefix and version. For example:
python -c 'import distutils.sysconfig; print
distutils.sysconfig.get_python_lib(plat_specific=1, standard_lib=1)'
Yields:
/usr/local/sys/python/lib/python2.3
On one Solaris platform I use. On Mac OS X it yields:
/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3
Indrek, can you see what this command yields for you and whether it
correctly yields what you need. Maybe it is a better of way of working
this out.
Graham
The distutils example above correctly yields /usr/lib64/python2.3/ on
the Opteron box.
OK, that looks like the way to go. I will make the correction to
configure.in and configure.
Jim
Re: mod_python 3.2.3b available for testing
Dominic Wong wrote: -1 for Gentoo Linux 2.6.13-gentoo-r3 Apache 2.0.54 Python 2.4.1 Hi Dominic, When you have a chance could you apply the following patch and re-run the tests? Thanks, Jim Index: test/test.py === --- test/test.py(revision 327731) +++ test/test.py(working copy) @@ -590,6 +590,7 @@ print \n * Testing req.headers_out conn = httplib.HTTPConnection(127.0.0.1:%s % PORT) +conn.set_debuglevel(1) conn.putrequest(GET, /, skip_host=1) conn.putheader(Host, test_req_headers_out:%s % PORT) conn.endheaders()
Re: mod_python 3.2.3b available for testing
Nick wrote: More info: python 2.4.2 on Linux: import tempfile t = tempfile.TemporaryFile() t open file 'fdopen', mode 'w+b' at 0xb7df07b8 type(t) type 'file' dir(t) ['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__', '__init__', '__iter__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', 'close', 'closed', 'encoding', 'fileno', 'flush', 'isatty', 'mode', 'name', 'newlines', 'next', 'read', 'readinto', 'readline', 'readlines', 'seek', 'softspace', 'tell', 'truncate', 'write', 'writelines', 'xreadlines'] python 2.4.1 on windows: import tempfile t = tempfile.TemporaryFile() t open file 'fdopen', mode 'w+b' at 0x0099FBA8 type(t) type 'instance' dir(t) ['__doc__', '__getattr__', '__init__', '__module__', '__repr__', 'close_called', 'file', 'name'] So this is an inconsistency within Python. Should mod_python attempt to correct it, or just claim a Python bug? I think we should correct it. I'm sure users don't care that we implement this with TemporaryFile. That being said, I wonder how many applications on Windows we may break by fixing this? Version 3.1.4 also used TemporaryFile, so this is not a new bug. Jim Nick wrote: This may be a Python Windows thing, but it shows up in mod_python: When using util.FieldStorage on multipart/form-data encoded POST data containing a file, in Linux a field.file will yield a file object (actually a subclass of file), but in Windows you have to get the file object through field.file.file. This probably has something to do with the fact that Windows' implementation of tempfile.TemporaryFile is different from Linux, but it should be made consistent in the mod_python interface. Nick
Re: mod_python 3.2.3b available for testing
Nick wrote: Jim Gallacher wrote: So this is an inconsistency within Python. Should mod_python attempt to correct it, or just claim a Python bug? I think we should correct it. I'm sure users don't care that we implement this with TemporaryFile. That being said, I wonder how many applications on Windows we may break by fixing this? Version 3.1.4 also used TemporaryFile, so this is not a new bug. Yeah, I never noticed it either until someone pointed it out to me. I appreciated the change to TemporaryFile, but being primarily a Linux user I never noticed that this broke my code in Windows. In any case, I'm still gonna have to implement a workaround in my own code to catch people using the different versions of mod_python out there, so I can live with whatever decision you guys make. But here's +1 for making the interface consistent at least for mod_python users. As for code breakage, I would consider this a bug introduced in 3.1.4, which was the last official release of mod_python, which will be corrected in release 3.3. You may have misunderstood. I was not suggesting that tempfile.TemporaryFile was introduced in 3.1.4, only that it existed there. Looking at the svn repository I see it's used in 3.0.0-beta and 2.7.9, so this bug has been lurking for a while. ;) Regards, Jim
Re: mod_python 3.2.3b available for testing
Indrek Järve wrote: This behaviour has been with Python for quite a while, so claiming it's simply a Python bug will be the same as declaring we don't support Windows. Our company's software that runs on Windows and uses mod_python simply patches util.py with the following change: 227c227 if isinstance(item.file, FileType): --- if isinstance(item.file, FileType) or (hasattr(item.file, 'file') and isinstance(item.file.file, FileType)): I haven't tried this with mp32 yet (we're still running on Python 2.3 and I haven't had time to investigate how to compile mp on Windows), but on 3.0/3.1 it appears to work just fine for our customers. The relevant part of FieldStorage has changed in 3.2. isinstance(item.file, FileType) or \ isinstance(getattr(item.file, 'file', None), FileType): so no more patching for you! Now I just need to understand what Nick is on about. ;) Jim Nick wrote: More info: python 2.4.2 on Linux: import tempfile t = tempfile.TemporaryFile() t open file 'fdopen', mode 'w+b' at 0xb7df07b8 type(t) type 'file' dir(t) ['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__', '__init__', '__iter__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', 'close', 'closed', 'encoding', 'fileno', 'flush', 'isatty', 'mode', 'name', 'newlines', 'next', 'read', 'readinto', 'readline', 'readlines', 'seek', 'softspace', 'tell', 'truncate', 'write', 'writelines', 'xreadlines'] python 2.4.1 on windows: import tempfile t = tempfile.TemporaryFile() t open file 'fdopen', mode 'w+b' at 0x0099FBA8 type(t) type 'instance' dir(t) ['__doc__', '__getattr__', '__init__', '__module__', '__repr__', 'close_called', 'file', 'name'] So this is an inconsistency within Python. Should mod_python attempt to correct it, or just claim a Python bug? Nick Nick wrote: This may be a Python Windows thing, but it shows up in mod_python: When using util.FieldStorage on multipart/form-data encoded POST data containing a file, in Linux a field.file will yield a file object (actually a subclass of file), but in Windows you have to get the file object through field.file.file. This probably has something to do with the fact that Windows' implementation of tempfile.TemporaryFile is different from Linux, but it should be made consistent in the mod_python interface. Nick
Re: mod_python 3.2.3b available for testing
Jorey Bump wrote:
Jim Gallacher wrote:
Nick wrote:
More info:
python 2.4.2 on Linux:
import tempfile
t = tempfile.TemporaryFile()
t
open file 'fdopen', mode 'w+b' at 0xb7df07b8
type(t)
type 'file'
dir(t)
['__class__', '__delattr__', '__doc__', '__getattribute__',
'__hash__', '__init__', '__iter__', '__new__', '__reduce__',
'__reduce_ex__', '__repr__', '__setattr__', '__str__', 'close',
'closed', 'encoding', 'fileno', 'flush', 'isatty', 'mode', 'name',
'newlines', 'next', 'read', 'readinto', 'readline', 'readlines',
'seek', 'softspace', 'tell', 'truncate', 'write', 'writelines',
'xreadlines']
python 2.4.1 on windows:
import tempfile
t = tempfile.TemporaryFile()
t
open file 'fdopen', mode 'w+b' at 0x0099FBA8
type(t)
type 'instance'
dir(t)
['__doc__', '__getattr__', '__init__', '__module__', '__repr__',
'close_called', 'file', 'name']
So this is an inconsistency within Python. Should mod_python attempt
to correct it, or just claim a Python bug?
I think we should correct it. I'm sure users don't care that we
implement this with TemporaryFile. That being said, I wonder how many
applications on Windows we may break by fixing this? Version 3.1.4
also used TemporaryFile, so this is not a new bug.
Are you sure there is anything to correct? In both cases, the object has
the same methods available for manipulating files (t.write('a'), for
example). They are not the same type of object, so they have different
dir() output, but don't they have the same functionality? What
specifically gets broken in util.FieldStorage?
No, I'm not sure. Now that I play around with it I'm not sure I
understand the problem at all. Perhaps Nick could elaborate?
Testing with python3.2.3 on Wine:
import tempfile
from types import *
t = tempfile.TemporaryFile()
t
open file 'fdopen', mode 'w+b' at 0x40D6A560
t.file
open file 'fdopen', mode 'w+b' at 0x40D6A560
t.write('stuff')
t.seek(0)
t.read()
isinstance(t, FileType)
False
Other than the fact that isinstance(t, FileType): returns False, I
don't see the problem. Nick?
Jim
Re: Gentoo (Was: mod_python 3.2.3b available for testing)
Gregory (Grisha) Trubetskoy wrote: If we don't get an resolution on this Gentoo issue - should we just go ahead and release the file anyway? Hopefully then someone will fix it before the final release? Since we have not received any additional information on this I think we should proceed. However there is also the issue of Too many cookies using mod_python 3.2.2b from a thread on the mod_python list. I don't know if we should generate another beta or just fix it in 3.2 final. The fix is very simple - remove 2 lines which had been added for the get_session feature, but that feature didn't make it into 3.2. There is also the issue of ./configure failing for x86-64 platforms. Again, it is a very simple fix - 3 lines in configure.in. I'll create JIRA issues and commit the fixes for these real soon. (tonight - maybe). To what extent can a final release differ from the last beta, Grisha? Is it OK to make some small changes from one to the other? When would changes be considered critical enough to require a new beta release round? Regards, Jim Grisha On Tue, 25 Oct 2005, Gregory (Grisha) Trubetskoy wrote: Hmmm... Looking at /usr/lib/python2.4/httplib.py, sock.readline() gets an EOF upon reading the first byte. Do you see anything in the error logs associated with this, like a segfault? To make it easier to isolate, try editing test.py to comment out all other tests and just leave this one. Look for a line that looks like: perRequestSuite.addTest(PerRequestTestCase(test_req_headers_out)) and comment out every other test in this block but this one, then rerun it. At the end you should have a log file in the logs directory, hopefully it will contain a clue. Thanks! On Tue, 25 Oct 2005, Dominic Wong wrote: -1 for Gentoo Linux 2.6.13-gentoo-r3 Apache 2.0.54 Python 2.4.1 * Testing req.headers_out connect: (127.0.0.1, 32873) send: 'GET / HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: test_req_headers_out:32873\r\n\r\n' reply: '' E == ERROR: test_req_headers_out (__main__.PerRequestTestCase) -- Traceback (most recent call last): File test.py, line 597, in test_req_headers_out response = conn.getresponse() File /usr/lib/python2.4/httplib.py, line 863, in getresponse response.begin() File /usr/lib/python2.4/httplib.py, line 333, in begin version, status, reason = self._read_status() File /usr/lib/python2.4/httplib.py, line 297, in _read_status raise BadStatusLine(line) BadStatusLine --
Re: [jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Graham Dumpleton (JIRA) wrote: [ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ] Graham Dumpleton updated MODPYTHON-77: -- Attachment: grahamd_20051105.tar.gz Here is my first go at an alternate patch for this problem. Patch was made against SVN head, believed to be 3.2.4b. All the change in effect does is save the first interpreter as main_interpreter, but most importantly it does this in Apache child process and not in the parent process before the fork occurs as was the case before. This avoids problems on Mac OS X where Apache would crash on restart and on Linux where Apache would crash after the request had been handled. Note that this change doesn't use any of the PEP GIL specific calls nor does it do anything specific to make anything work on Python 2.3.5. Except for moving one thread state swap call from the parent process context to the child process context, all thread management code is the same. The changes work fine on: Mac OS X (10.3.9) / Apache 2.0.51 (worker) / Python 2.3 (Apple OS Installed) Linux Fedora Code 2 / Apache 2.0.55 (prefork) / Python 2.3.5 Test example was gilstate.tar.gz attached to MODPYTHON-77. Also passed on mod_python/test suite on Mac OS X. There were failures of test suite on Linux, but those failures occurred before patches were applied as well. Failures? What failures? Jim
Re: mod_python.util.StorageField.read_to_boundary has problems in 3.1 and 3.2
Alexis,
Do you a have a small file which shows this behaviour and could be used
for testing? Even better would be a function which would generate a test
file. This could be included in the mod_python unit tests.
Jim
Alexis Marrero wrote:
All,
The current 3.1 mod_python implementation of
mod_python.util.StorageField.read_to_boudary reads as follows:
203 def read_to_boundary(self, req, boundary, file):
204 delim =
205 line = req.readline()
206 sline = line.strip()
207 last_bound = boundary + --
208 while line and sline != boundary and sline != last_bound:
209 odelim = delim
210 if line[-2:] == \r\n:
211 delim = \r\n
212 line = line[:-2]
213 elif line[-1:] == \n:
214 delim = \n
215 line = line[:-1]
216 file.write(odelim + line)
217 line = req.readline()
218 sline = line.strip()
As we have discussed previously:
http://www.modpython.org/pipermail/mod_python/2005-March/017754.html
http://www.modpython.org/pipermail/mod_python/2005-March/017756.html
http://www.modpython.org/pipermail/mod_python/2005-November/019460.html
This triggered couple of changes in mod_python 3.2 Beta which reads as
follows:
33 # Fixes memory error when upload large files such as 700+MB ISOs.
34 readBlockSize = 65368
35
*...*
225 def read_to_boundary(self, req, boundary, file):
...
234 delim = ''
235 lastCharCarried = False
236 last_bound = boundary + '--'
237 roughBoundaryLength = len(last_bound) + 128
238 line = req.readline(readBlockSize)
239 lineLength = len(line)
240 if lineLength roughBoundaryLength:
241 sline = line.strip()
242 else:
243 sline = ''
244 while lineLength 0 and sline != boundary and sline !=
last_bound:
245 if not lastCharCarried:
246 file.write(delim)
247 delim = ''
248 else:
249 lastCharCarried = False
250 cutLength = 0
251 if lineLength == readBlockSize:
252 if line[-1:] == '\r':
253 delim = '\r'
254 cutLength = -1
255 lastCharCarried = True
256 if line[-2:] == '\r\n':
257 delim += '\r\n'
258 cutLength = -2
259 elif line[-1:] == '\n':
260 delim += '\n'
261 cutLength = -1
262 if cutLength != 0:
263 file.write(line[:cutLength])
264 else:
265 file.write(line)
266 line = req.readline(readBlockSize)
267 lineLength = len(line)
268 if lineLength roughBoundaryLength:
269 sline = line.strip()
270 else:
271 sline = ''
This function has a mysterious bug in it... For some files which I could
disclose (one of them been the PDF file for Apple's Pages User Manual in
Italian) the uploaded file in the server ends up with the same length
but different sha512 (the only digest that I'm using). The problem is a
'\r' in the middle of a chunk of data that is much larger
than readBlockSize.
Anyhow, I wrote a new function, which I believe is much simpler, and
test it with thousands and thousands of different files and so far it
seems to work fine. It reads as follows:
def read_to_boundary(self, req, boundary, file):
''' read from the request object line by line with a maximum size,
until the new line starts with boundary
'''
previous_delimiter = ''
while 1:
line = req.readline(116)
if line.startswith(boundary):
break
if line.endswith('\r\n'):
file.write(previous_delimiter + line[:-2])
previous_delimiter = '\r\n'
elif line.endswith('\r') or line.endswith('\n'):
file.write(previous_delimiter + line[:-1])
previous_delimiter = line[-1:]
else:
file.write(previous_delimiter + line)
previous_delimiter = ''
Let me know any comments on it and if you test it and fails please also
let me know. I don't have subversion account neither I don't know how to
use it thus this email.
/amn
___
Mod_python mailing list
[EMAIL PROTECTED] mailto:[EMAIL PROTECTED]
http://mailman.modpython.org/mailman/listinfo/mod_python
Re: mod_python.util.StorageField.read_to_boundary has problems in 3.1 and 3.2
I've been spending some quality time with hexedit, vim and a little bit
of python. I can now generate a file which can be used in the unit test.
The problem seems to occur when a '\r' character is right at
readBlockSize boundary, which is 65368 in the current mod_python.util.
I have not yet tested Alexis's fix yet. It's possible that it doesn't
fix the problem but is just masking it since his code uses a
readBlockSize of 65536, which will not be a problem for ugh.pdf.
The attached script will generate a file that can be used for testing. I
haven't tried to integrate it into Nicolas's unit test as I wanted to
share my findings with everyone asap. I'll spend some more time on this
tommorrow.
Happy Bug Hunting,
Jim
Nicolas Lehuen wrote:
Hi guys,
In the pure if it ain't tested, it ain't fixed fashion, I've added a
unit test for file upload to the test suite. It uploads a randomly
generated 1 MB file to the server, and check that the MD5 digest
returned by the server is correct. I could not reproduce Alexis' bug
report this way, however. But I then I added a test with the UNIX-HATERS
handbook file ugh.pdf, and bang, here comes the bug.
I've checked in both unit tests into subversion, so that you can play
with them. I'm now going to test Alexis' fix.
#!/usr/bin/env python
import sys
def generate_file(offset=-1, readBlockSize=65368, fname='testfile'):
Generate a file which causes the error with file upload
The default offset of -1 should generate a file which will
be corrupted by the file upload.
Test Results:
offset = -2
235a26d377f0b49ba2fd8706f27dc9e5 testfile.-2.bin
235a26d377f0b49ba2fd8706f27dc9e5 testfile.-2.bin.res
offset = -1
** THIS ONE FAILS **
39626fae4bd0812a2522ebe45c23e186 testfile.-1.bin
e1fc1008bfa9a98dde6c9481e3c83c43 testfile.-1.bin.res
offset = 0
8bd673f325f250a90d623defe1fff11d testfile.0.bin
8bd673f325f250a90d623defe1fff11d testfile.0.bin.res
offset = 1
d0f4adf904cdc058f0ac7013baa83998 testfile.1.bin
d0f4adf904cdc058f0ac7013baa83998 testfile.1.bin.res
# readBlockSize is the same as the value from mod_python.util
f = open('%s.%d.src' % (fname, offset), 'w')
f.write('a'*100)
f.write('\r\n')
block_size = readBlockSize + offset
f.write('b'*block_size)
f.write('\rccc')
f.write('d'*100)
f.write('\r\n')
f.close()
if __name__ == '__main__':
generate_file(offset=int(sys.argv[1]))
Re: mod_python.util.StorageField.read_to_boundary has problems in 3.1 and 3.2
Gregory (Grisha) Trubetskoy wrote: So I guess this means we roll and vote on a 3.2.5b? As much as it pains me to say it, but yes, this is a must fixm so it's on to 3.2.5b. I think we need to do some more extensive testing on Alexis's fix before we roll 3.2.5b. His read_to_boundary is much simpler than the current one. This makes me wonder if there is some magic happening in the current version which is solving some weird problems, or is his code just that much better? I'm feeling a little dull right now so all the code just blurs together. It also doesn't help that util.py uses *3 spaces* for the indent. Yikes. How the heck did that happen? :( I'll take another look tomorrow. Jim
Re: mod_python.util.StorageField.read_to_boundary has problems in 3.1 and 3.2
Alexis Marrero wrote: Sorry for all this emails, No worries. It's a bug that needs to be fixed, so your work will benefit everyone. :) Jim
Re: mod_python.util.StorageField.read_to_boundary has problems in 3.1 and 3.2
Alexis Marrero wrote:
Ok. Now I'm confused.
So am I!
I've created a test harness so we can bypass mod_python completely. It
includes a slightly modified version of read_to_boundary which adds a
new parameter, readBlockSize.
In the output from the test harness, your version is 'new' and the
current version is 'cur'. Run it and see what happens.
Jim
$ ./upload_test_harness
generate_embedded_cr_file
test offset -1 chunk []
src 5a63347d1106afdfa264b2a61f81ae82
cur 5a63347d1106afdfa264b2a61f81ae82 PASS
new 5a63347d1106afdfa264b2a61f81ae82 PASS
test offset -1 chunk ['CR']
src 82204e52343d5b25c2e783cd59499973
cur e4af2eee73029642a114697ba59217b3 FAIL
new 82204e52343d5b25c2e783cd59499973 PASS
generate_split_boundary_file
test offset -1 chunk []
src d481990a0f0bbd8acf847cd732714555
cur d481990a0f0bbd8acf847cd732714555 PASS
new 8fa5ac9f913d778575ea871506c392a9 FAIL
test offset -1 chunk ['CR']
src 8fa5ac9f913d778575ea871506c392a9
cur d481990a0f0bbd8acf847cd732714555 FAIL
new 8fa5ac9f913d778575ea871506c392a9 PASS
What I was trying to say is that I created a file with this function:
def generate_split_file(offset=-1,
readBlockSize=65368,
fname='testfile'):
f = open(fname, 'w')
f.write('a'*50)
f.write('\r\n')
block_size = readBlockSize + offset
f.write('b'*block_size)
f.close()
Then I uploaded 'testfile' using the following
StorageField.read_to_boundary() method:
def read_to_boundary(self, req, boundary, file):
''' read from the request object line by line with a maximum size,
until the new line starts with boundary
'''
previous_delimiter = ''
while 1:
line = req.readline(116)
if line.startswith(boundary):
break
if line.endswith('\r\n'):
file.write(previous_delimiter + line[:-2])
previous_delimiter = '\r\n'
elif line.endswith('\r') or line.endswith('\n'):
file.write(previous_delimiter + line[:-1])
previous_delimiter = line[-1:]
else:
file.write(previous_delimiter + line)
previous_delimiter = ''
And the md5 in the client is the same one as in the server. Do you
have different results? Let me know.
Regards,
/amn
On Nov 7, 2005, at 2:11 PM, Jim Gallacher wrote:
Jim Gallacher wrote:
Alexis Marrero wrote:
Jim,
Thanks for sending the function that creates the test file. However
I ran it to create the test file, and after uploading the file the
MD5 still the same.
Just to clarify, is this for your new read_to_boundary or the one in
3.2? If it's for yours then the MD5 sum *should* be the same, since
that's what you fixed. :)
Did you call it with the same block size as you are using in your
code? The '\r' character must appear in the file right at the
readBlockSize boundary.
ie.
generate_file(offset=-1, readBlockSize=116, fname='testfile')
#!/usr/bin/env python
from mkfile import generate_split_file, generate_file
import sys
from StringIO import StringIO
import md5
def read_to_boundary_current(self, req, boundary, file, readBlockSize):
''' currrent version '''
#
# Although technically possible for the boundary to be split by the read, this will
# not happen because the readBlockSize is set quite high - far longer than any boundary line
# will ever contain.
#
# lastCharCarried is used to detect the situation where the \r\n is split across the end of
# a read block.
#
delim = ''
lastCharCarried = False
last_bound = boundary + '--'
roughBoundaryLength = len(last_bound) + 128
line = req.readline(readBlockSize)
lineLength = len(line)
if lineLength roughBoundaryLength:
sline = line.strip()
else:
sline = ''
while lineLength 0 and sline != boundary and sline != last_bound:
if not lastCharCarried:
file.write(delim)
delim = ''
else:
lastCharCarried = False
cutLength = 0
if lineLength == readBlockSize:
if line[-1:] == '\r':
delim = '\r'
cutLength = -1
lastCharCarried = True
if line[-2:] == '\r\n':
delim += '\r\n'
cutLength = -2
elif line[-1:] == '\n':
delim += '\n'
cutLength = -1
if cutLength != 0:
file.write(line[:cutLength])
else:
file.write(line)
line = req.readline(readBlockSize)
lineLength = len(line
Re: [jira] Created: (MODPYTHON-87) psp_parser: replaces \n on \n
I'll give it a try, and create a unit test at the same time. Should the
unit tests cover other possibilites such as \t, \r and so on?
Jim
Gregory (Grisha) Trubetskoy wrote:
I think the fix to that may be inserting
TEXT\\n {
psp_string_appendl(PSP_PG(pycode), STATIC_STR(n));
}
into psp_parser.l - could someone try it?
The explanation is that it looks like compile() treats '\n' specially.
BTW, there may be other sequnces that it treats specially, e.g. \t ?
s = 'print TEST\n\\n\nTEST2\n\n\n'
c = compile(s, blah, exec)
c
code object ? at 0xb7ce5a20, file blah, line 1
exec c
TEST
TEST2
s2 = 'print TEST\nn\nTEST2\n\n\n'
c2 = compile(s2, blah, exec)
exec c2
TEST
\n
TEST2
Grisha
On Tue, 8 Nov 2005, Anton Kuzmin (JIRA) wrote:
LF character
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
psp_parser: replaces \n on \n
LF character
-
Key: MODPYTHON-87
URL: http://issues.apache.org/jira/browse/MODPYTHON-87
Project: mod_python
Type: Bug
Versions: 3.2
Environment: Debian unstable, mod_python/3.2.0b Python/2.3.5
Reporter: Anton Kuzmin
$ cat mptest.psp
html
body
You see (#92;n) in the test result. If you see () then the test
fails.br /
BEGIN TESTbr /
(\n)
br /END TEST
/body
/html
The result on the screen is:
You see (\n) in the test result. If you see () then the test fails.
BEGIN TEST
( )
END TEST
$ cat .htaccess
AddHandler mod_python .psp
PythonHandler mod_python.psp
PythonDebug On
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: mod_python 3.2.5b available for testing
+1 Linux Debian 3.1 stable (sarge) apache 2.0.54-5 (mpm-worker) python 2.3.5 gcc 3.3.5 +1 Linux Debian unstable (sid) apache 2.0.54-4 (mpm-prefork) python 2.3.5 gcc 4.0.2
Re: mod_python 3.2.5b available for testing (gentoo issues)
+1 with patch Linux gentoo 2.6.12-gentoo-r6 apache 2.0.54 (mpm-prefork) python 2.4.2 gcc 3.3.6 There are 2 issues with the unit tests in Gentoo that are fixed by the attached patch. (Just to be clear, I mean the problems are with the unit test code, not with mod_python). First, test_global_lock uses ab in the test. In Gentoo, ab is named ab2, so this test fails. The attached patch just skips the test if ab doesn't exist. We can improve on this later. The second issue is with test_req_headers_out, first reported by Dominic Wong a couple of weeks ago. The fix is simple, but I want to understand why it was failing under Gentoo and not other platforms. The culprit seems to be the use of DirectoryIndex directive in the test.conf. The existence of DirectoryIndex /tests.py causes apache to segfault. Removing DirectoryIndex and giving the full url in the putrequest allows the test to complete successfully. So my questions are: 1. Why would DirectoryIndex cause a segfault on gentoo but not other platforms? 2. This test is the only one that uses DirectoryIndex. Is there any special reason for this? 3. This test is the only one that uses AddHandler instead of SetHandler. Is there a reason for this? 4. This test is the only one that sets a PythonAccessHandler directive in test.conf. Is there a reason for this? Can anyone offer any insights? Regards, Jim Index: test/test.py === --- test/test.py(revision 344202) +++ test/test.py(working copy) @@ -136,7 +136,6 @@ MOD_PYTHON_SO = testconf.MOD_PYTHON_SO LIBEXECDIR = testconf.LIBEXECDIR AB = os.path.join(os.path.split(HTTPD)[0], ab) - SERVER_ROOT = TESTHOME CONFIG = os.path.join(TESTHOME, conf, test.conf) DOCUMENT_ROOT = os.path.join(TESTHOME, htdocs) @@ -379,7 +378,6 @@ return rsp ### Tests begin here - def test_req_document_root_conf(self): c = VirtualHost(*, @@ -668,8 +666,7 @@ ServerName(test_req_headers_out), DocumentRoot(DOCUMENT_ROOT), Directory(DOCUMENT_ROOT, - AddHandler(mod_python .py), - DirectoryIndex(/tests.py), + SetHandler(mod_python), PythonHandler(tests::req_headers_out), PythonAccessHandler(tests::req_headers_out_access), PythonDebug(On))) @@ -680,7 +677,7 @@ print \n * Testing req.headers_out conn = httplib.HTTPConnection(127.0.0.1:%s % PORT) -conn.putrequest(GET, /, skip_host=1) +conn.putrequest(GET, /tests.py, skip_host=1) conn.putheader(Host, test_req_headers_out:%s % PORT) conn.endheaders() response = conn.getresponse() @@ -1733,6 +1730,10 @@ t1 = time.time() print , time.ctime() ab = quoteIfSpace(AB) +if not os.path.exists(ab): +print ab not found. Skipping _global_lock test +return + if os.name == nt: cmd = '%s -c 5 -n 5 http://127.0.0.1:%s/tests.py NUL:' \ % (ab, PORT)
Re: mod_python 3.2.5b available for testing (gentoo issues)
Nicolas Lehuen wrote: Hi Jim, 2005/11/16, Jim Gallacher [EMAIL PROTECTED]: +1 with patch Linux gentoo 2.6.12-gentoo-r6 apache 2.0.54 (mpm-prefork) python 2.4.2 gcc 3.3.6 There are 2 issues with the unit tests in Gentoo that are fixed by the attached patch. (Just to be clear, I mean the problems are with the unit test code, not with mod_python). First, test_global_lock uses ab in the test. In Gentoo, ab is named ab2, so this test fails. The attached patch just skips the test if ab doesn't exist. We can improve on this later. My bad, I'm the one who put back this test... It was commented out before mentioning a bug in ab with AFAIK the test ran perfectly. I didn't think about ab being renamed ab2... FYI, on debian ab2 is a symlink to ab. The second issue is with test_req_headers_out, first reported by Dominic Wong a couple of weeks ago. The fix is simple, but I want to understand why it was failing under Gentoo and not other platforms. The culprit seems to be the use of DirectoryIndex directive in the test.conf. The existence of DirectoryIndex /tests.py causes apache to segfault. Removing DirectoryIndex and giving the full url in the putrequest allows the test to complete successfully. So my questions are: 1. Why would DirectoryIndex cause a segfault on gentoo but not other platforms? I don't know why, but isn't it strange to put a slash in front of tests.py ? Shouldn't the directive be just DirectoryIndex tests.py N I tried using 'tests.py' as well, and it still segfaults. Weird. 2. This test is the only one that uses DirectoryIndex. Is there any special reason for this? 3. This test is the only one that uses AddHandler instead of SetHandler. Is there a reason for this? 4. This test is the only one that sets a PythonAccessHandler directive in test.conf. Is there a reason for this? Can anyone offer any insights? It also uses httplib and not vhost_get. It's as if this test was one of the first that have been written, and that ways to write better tests have been improved since (using vhost_get etc.). It makes sense to use httplib directly since it needs access to the headers. I did wonder if this was an early test though, and thus the different pattern used for the config. To be consistent with the other tests I think I'll remove the PythonAccessHandler from test_req_document_root_conf as well, unless you can see a reason it should stay. Jim
Re: mod_python 3.2.5b available for testing
Gregory (Grisha) Trubetskoy wrote: OK, I think we got enough +1's and no show-stopping problems (for a beta at least). I copied it over the apache server, once the mirrors sync I'll update the site and send the big announcement. I was also thinking it was time for a wider release, but was hoping to see a +1 for FreeBSD 4 or 5 first. I've been sporadically trying to test mod_python on FreeBSD 5.4 using qemu but with no success. Not a big suprise since I can't get 3.1.4 to work either. I've likely messed something up as this was my first attempt with either FreeBSD or qemu. FreeBSD feels like a foreign country where everyone speaks English but with a funny accent. I'm sure I'll adjust. On the subject of qemu, one of my longer term projects is to setup an automated test environment using qemu images of the most popular linux and bsd distros. I get to decide what's popular. No arguments. No flamewars. ;) Qemu ain't fast, but it is effective and cheap. (To put things in context the unit tests alone take approx 10 minutes. Granted my computer is not a screamer, but still...) So far I have working images of Debian stable, Debian unstable and Gentoo, plus the afore mentioned but messed up FreeBSD. As time (and disk space) permit I'll add some combination of Fedora, maybe one of the RHEL clones, SUSE, Ubuntu or whatever else strikes my fancy. Once I cobble together some scripts to control qemu and automate the configure/make/test cycle we'll have ourselves a nice little virtual build farm. Let the computers work while we sleep, I say. If I ever get around to fully implementing this grand scheme we'll have a tool which will hopefully speed up the release cycle. Is this Friday? It sure feels like Friday. Usually I only ramble on like this on Fridays. :) Dang. It's not Friday. Back to work. Jim
Re: [DRFT] ANNOUNCE: Mod_python 3.2.5 Beta
Grisha, Looks good and the links work. Were you planning on making an announcement (on the mod_python list at least) regarding your ApacheCon talk as well? Jim Gregory (Grisha) Trubetskoy wrote: Let me know if you see anything wrong in the text below, I'll send it out later today or tomorrow. The download page has been updated to show 3.2.5b already. -- Forwarded message -- From: Gregory (Grisha) Trubetskoy [EMAIL PROTECTED] To: announce@httpd.apache.org, [EMAIL PROTECTED] Cc: python-dev@httpd.apache.org Newsgroups: comp.lang.python Subject: ANNOUNCE: Mod_python 3.2.5 Beta The Apache Software Foundation and The Apache HTTP Server Project are pleased to announce the 3.2.5 Beta release mod_python. Version 3.2.5b of mod_python features several new functions and attributes providing better access to apache internals, file-based sessions and other session improvements, as well as many bug fixes and various performance and security improvements. A detailed description of the changes is available in Appendix A of the mod_python manual, also available here: http://www.modpython.org/live/mod_python-3.2.5b/doc-html/node97.html Beta releases are NOT considered stable and usually contain bugs. This release is intended to solicit widespread testing of the code. We strongly recommend that you try out your existing applications and experiment with new features in a non-production environment using this version and report any problems you may encounter so that they can be addressed before the final release. Preferred method of reporting problems is the mod_python user list [EMAIL PROTECTED] Mod_python 3.2.5b is available for download from: http://httpd.apache.org/modules/python-download.cgi For more information about mod_python visit http://www.modpython.org/ Regards, Grisha Trubetskoy and the Apache mod_python team.
Re: 3.2b5 : mod_python doesn't handle properly signals likeKILL,SEGV...
Michel Jouvin wrote: Graham, I played a little bit with worker MPM parameters. In particular I tested your suggestion to increase to 2 StartServers. This has no effect on the problem. I also tried to raise MaxSpareThread to MaxClient and suppressed child recycling (MaxRequestPerChild=0) to suppress restart of child as it seems to trig the problem with mod_pyton. No effect. I also checked the load during all these tests. Almost no request. So the heavy load syndroma you described doesn't seem to apply in this case. Again, one month ago I tested during 2 or 3 days an Apache configuration with mod_python loaded and without any url to trig its usages. And the problem was already the same. So it seems this is not related to mod_python usage (it happens even if you didn't execute any Python code) but rather to mod_python interaction with other Apache components. Michel Michel, I'm not able to reproduce the behaviour on debian stable (i386) with apache 2.0.54, but I'm not sure if I'm testing this correctly. Could you create a test script (bash or python) that will produce the error? That way I can know for sure that I'm testing in the same way. Jim
Re: 3.2b5 : mod_python doesn't handle properly signals likeKILL,SEGV...
Michel, I can't reproduce the problem on debian i386. I put together a script that continually greps a apache child pid and kills it. After killing 200 processes there is no change in the total number of apache processes, and nothing in the apache log other an entry for each process killed: [Thu Nov 24 17:03:44 2005] [error] cgid daemon process died, restarting ... Regards, Jim Michel Jouvin wrote: I don't know If really need to write a script, this is so simple. asa/root % ps -e -opid,ppid,cmd | grep http 15601381048577 /www/Web/servers/apache/2.0.54/bin/httpd -k start 15601631560138 /www/Web/servers/apache/2.0.54/bin/httpd -k start 10863961086105 grep http From this output, you see that 1560163 is the child. Kill it with : kill -KILL 1560163 If you enter again 'ps -e|grep http', you'll see (I am seeing...) the number of httpd processes increasing until the max number (determined by MaxClient and ThreadPerChild). When this max number is reached you get the error message in main Apache error log. Michel --On mercredi 23 novembre 2005 19:30 -0500 Jim Gallacher [EMAIL PROTECTED] wrote: Michel Jouvin wrote: Graham, I played a little bit with worker MPM parameters. In particular I tested your suggestion to increase to 2 StartServers. This has no effect on the problem. I also tried to raise MaxSpareThread to MaxClient and suppressed child recycling (MaxRequestPerChild=0) to suppress restart of child as it seems to trig the problem with mod_pyton. No effect. I also checked the load during all these tests. Almost no request. So the heavy load syndroma you described doesn't seem to apply in this case. Again, one month ago I tested during 2 or 3 days an Apache configuration with mod_python loaded and without any url to trig its usages. And the problem was already the same. So it seems this is not related to mod_python usage (it happens even if you didn't execute any Python code) but rather to mod_python interaction with other Apache components. Michel Michel, I'm not able to reproduce the behaviour on debian stable (i386) with apache 2.0.54, but I'm not sure if I'm testing this correctly. Could you create a test script (bash or python) that will produce the error? That way I can know for sure that I'm testing in the same way. Jim * * Michel Jouvin Email : [EMAIL PROTECTED] * * LAL / CNRSTel : +33 1 64468932* * B.P. 34 Fax : +33 1 69079404* * 91898 Orsay Cedex * * France* *
Re: 3.2b5 : mod_python doesn't handle properly signals likeKILL,SEGV...
Michel Jouvin wrote: Jim, I am not totally surprised... I am afraid this is a platform specific issue as we are running mod_python on Tru64. Something like a 64 bits issue. Does it sound a reasonnable possibility ? I have no idea what may be going on, but that seems as likely as anything else. How to progress in troubleshooting ? Again, no clue. :(. Hopefully some of the bigger brains that hang out around here will chime in. I know Indrek Järve tested 3.2.2b on SuSE Linux 9.2 (x86-64). Perhaps he or someone else with a 64-bit platform could try and reproduce the problem. That would tell us if it's 64-bit related or Tru64 related. I've attached my test script if anyone wants to mess with it. I'm sure I don't need to tell you to *not* run it on a production machine. ;) You'll likely want to change the PAUSE variable to something less than 30 seconds, which is the time between the kill calls. I was testing using qemu, and it needs lots of time for things to happen. usage: ./killchildren # number of loops Jim Michel --On jeudi 24 novembre 2005 17:41 -0500 Jim Gallacher [EMAIL PROTECTED] wrote: Michel, I can't reproduce the problem on debian i386. I put together a script that continually greps a apache child pid and kills it. After killing 200 processes there is no change in the total number of apache processes, and nothing in the apache log other an entry for each process killed: [Thu Nov 24 17:03:44 2005] [error] cgid daemon process died, restarting ... Regards, Jim Michel Jouvin wrote: I don't know If really need to write a script, this is so simple. asa/root % ps -e -opid,ppid,cmd | grep http 15601381048577 /www/Web/servers/apache/2.0.54/bin/httpd -k start 15601631560138 /www/Web/servers/apache/2.0.54/bin/httpd -k start 10863961086105 grep http From this output, you see that 1560163 is the child. Kill it with : kill -KILL 1560163 If you enter again 'ps -e|grep http', you'll see (I am seeing...) the number of httpd processes increasing until the max number (determined by MaxClient and ThreadPerChild). When this max number is reached you get the error message in main Apache error log. Michel --On mercredi 23 novembre 2005 19:30 -0500 Jim Gallacher [EMAIL PROTECTED] wrote: Michel Jouvin wrote: Graham, I played a little bit with worker MPM parameters. In particular I tested your suggestion to increase to 2 StartServers. This has no effect on the problem. I also tried to raise MaxSpareThread to MaxClient and suppressed child recycling (MaxRequestPerChild=0) to suppress restart of child as it seems to trig the problem with mod_pyton. No effect. I also checked the load during all these tests. Almost no request. So the heavy load syndroma you described doesn't seem to apply in this case. Again, one month ago I tested during 2 or 3 days an Apache configuration with mod_python loaded and without any url to trig its usages. And the problem was already the same. So it seems this is not related to mod_python usage (it happens even if you didn't execute any Python code) but rather to mod_python interaction with other Apache components. Michel Michel, I'm not able to reproduce the behaviour on debian stable (i386) with apache 2.0.54, but I'm not sure if I'm testing this correctly. Could you create a test script (bash or python) that will produce the error? That way I can know for sure that I'm testing in the same way. Jim * * Michel Jouvin Email : [EMAIL PROTECTED] * * LAL / CNRSTel : +33 1 64468932* * B.P. 34 Fax : +33 1 69079404* * 91898 Orsay Cedex * * France* * * * Michel Jouvin Email : [EMAIL PROTECTED] * * LAL / CNRSTel : +33 1 64468932* * B.P. 34 Fax : +33 1 69079404* * 91898 Orsay Cedex * * France* * killchildren.sh Description: application/shellscript
Re: More efficient StringField and Field classes
Hi Mike, I don't have time to dig into this tonight, but your patch causes one of the unit tests to fail. FAIL: test_util_fieldstorage (__main__.PerRequestTestCase) -- Traceback (most recent call last): File test.py, line 1117, in test_util_fieldstorage self.fail(`rsp`) File /usr/lib/python2.3/unittest.py, line 270, in fail raise self.failureException, msg AssertionError: ['1', '2', '3', '4'] Jim
Re: [mod_python] ANNOUNCE: Mod_python 3.2.5 Beta
Grisha, Speaking of 3.2.5 beta, how long do we wait before it becomes final? Jim
Re: [jira] Commented: (MODPYTHON-93) Improve util.FieldStorage efficiency
Gregory (Grisha) Trubetskoy wrote:
Having looked at the FieldStorage code, I'm guessing the idea was that
you parse fields as they come in and append them to a list. This
preserves the original order of fields, in case it is needed.
I assumed that as well, but I'm not sure getting the fields in a
particular order is the most common use case - not for me anyway. Plus,
I'm not suggesting getting rid of access to FieldStorage.list. For the
dict-like such as fs.keys(), I don't think people should expect keys in
any particular order. If they really want that feature then they can
subclass write their own ordered dict class.
I'm not sure that maintaining a dictionary alongside the list is the
right thing to do. It might be, but there are some difficult questions
to answer -e.g. how costly is a sequential search, and is the code
complexity (and fieldstorage code is no picnic to read as it is) worth
the speedup?
The current code is a litte hairy - it helps to drink a cup of strong
coffee before reading it. We can always hide the complexity in a
separate add_field method. As to the performance tradeoffs, I guess we
benchmark and see? I love doing benchmarks. The 2 things that bring joy
to my heart are benchmarks and unit tests. And good documentation. The
*3* things that bring joy to my heart are... well you get the idea.
Also while it would speed up retrieval, it will slow down the write
operation - when a field is added to fieldstorage you now need to append
it to the list, AND check whether it exists in the dictionary, then add
it there as well.
How often do developers access form fields via __getitem__? I noticed
the publisher does not use it - it iterates the list, so nothing would
be gained there.
For myself, I use it (almost?) exclusively as a dict. As for the use in
publisher, maybe that implementation needs to be examined as well. ;)
Also, something else to consider - is there a simple programatic
solution that could be documented, e.g. something like
my_fs = util.FieldStorage(req)
dict_fs = {}
dict_fs.update(my_fs)
[have no idea whether this will work :-)]
Nope. If you have multiple fields with the same name you'll lose all but
the last field. (eg. The checkbox example example on the mod_python list
that got me started on this in the first place).
and voila - you've got a dictionary based fieldstorage?
Except that FieldStorage is already supposed to be dict-like so why
would I want to duplicate the effort in my code? For example 7 out of 10
the fs methods are there to support dict-like behaviour and the other
3 are initialization helpers which will never be called by user code
anyway. To me, that makes it a dictionary. I'm not talking about adding
new functionality to FieldStorage, just examining the current
implementation wrt to performance.
Anyway, just a few cents from me.
I don't want you to think I'm hung up on this issue. It just seems to me
that the goal of mod_python moving forward should be stability, speed,
efficiency while keeping feature creep to a minumum. I think it's
worthwhile to examine existing code as we go along to see if we are
meeting these goals. We still need to have *some* code to chew on every
now and then after all. :)
Jim
Re: [mod_python] ANNOUNCE: Mod_python 3.2.5 Beta
Gregory (Grisha) Trubetskoy wrote: A couple of weeks perhaps? I don't think the final can happen before Apachecon without feeling rushed anyway, so we could target second half of December? Sounds good. But are you not at least a little bit tempted to have it have it ready to go so that you can make the official announcement at ApacheCon? Jim On Mon, 28 Nov 2005, Jim Gallacher wrote: Grisha, Speaking of 3.2.5 beta, how long do we wait before it becomes final? Jim
Re: [jira] Commented: (MODPYTHON-93) Improve util.FieldStorage efficiency
Gregory (Grisha) Trubetskoy wrote: On Mon, 28 Nov 2005, Nicolas Lehuen wrote: Why is the ordering so important ? I do understand we need to support multiple values per field name, but I don't see why ordering is needed. I think that it may be dictated by some RFC (the stdlib does it this way too), I'm not sure, but it's a good question though, it'd be great to have it researched and answered so that we don't have to go over this point again. Grisha Ordering is not defined according to my interpretation. But at the same time we shouldn't mess with the ordering. Gotta love those RFCs. :) http://www.ietf.org/rfc/rfc2388.txt?number=2388 Returning Values from Forms: multipart/form-data 5.5 Ordered fields and duplicated field names The relationship of the ordering of fields within a form and the ordering of returned values within multipart/form-data is not defined by this specification, nor is the handling of the case where a form has multiple fields with the same name. While HTML-based forms may send back results in the order received, and intermediaries should not reorder the results, there are some systems which might not define a natural order for form fields. Jim
Re: Various musings about the request URL / URI / whatever
Nicolas Lehuen wrote:
Hi,
Is it me or is it quite tiresome to get the URL that called us, or get
the complete URL that would call another function ?
When performing an external redirect (using mod_python.util.redirect for
example), we MUST (as per RFC) provide a full URL, not a relative one.
Instead of util.redirect(req,'/foo/bar.py'), we should write
util.redirect(req,'https://whatever:8443/foo/bar.py').
The problem is, writing this is always tiresome, as it means building a
string like this :
def current_url(req):
req.add_common_vars()
current_url = []
# protocol
if req.subprocess_env.get('HTTPS') == 'on':
current_url.append('https')
default_port = 443
else:
current_url.append('http')
default_port = 80
current_url.append('://')
# host
current_url.append(req.hostname)
# port
port = req.connection.local_addr[1]
if port != default_port:
current_url.append(':')
current_url.append(str(port))
# URI
current_url.append(req.uri)
return ''.join(current_url)
So I have two questions :
First question, is there a simpler way to do this ? Ironically, when
using mod_rewrite, you get an environment variable named SCRIPT_URI
which is precisely what I need (SCRIPT_URL, also added by mod_rewrite,
is equivalent to req.uri... Don't ask we why). But relying on it isn't
safe since mod_rewrite isn't always used.
I guess you could just assemble the parts from the req.parsed_uri tuple,
except that apache doesn't actually fill in parsed_uri. :(
Second question, if there isn't any simpler way to do this, should we
add it to mod_python ? Either as a function like above in
mod_python.util, or as a member of the request object (named something
like url to match the other member named uri, but that's just teasing).
I'm not against it, but for my purposes I think it would be more useful
for parsed_uri to just work properly.
And third question (in pure Spanish inquisition style) : why is
req.parsed_uri returning me a tuple full of Nones except for the uri and
path_info part ?
It comes from apache that way. I sure don't know why though. Maybe we're
missing some magic apache call that would fill it in?
Ah, fourth question : why are we (mod_python, mod_rewrite and the CGI
environment variables) using the terms URI and URL to distinguish
between a full, absolute resource path and a path relative to the
server, whereas the definition of URLs and URIs is very vague and
nothing close to this
(http://www.w3.org/TR/uri-clarification/#contemporary) ? Shouldn't we
save our souls and a lot of saliva by choosing better names ?
Strangely I was reading the cited page just last week, for perhaps the
100th time. I keep hoping I'll find enlightment but alas no. The danger
of choosing new names (ie absolute_thingy or relative_thingy) is that
we also add another layer of confusion. I'm not saying new names are a
bad idea, just that we need to be very careful.
OK, OK, fifth question : we made req.filename and other members
writable. But when those attributes are changed, as Graham noted a while
ago, the other dependent ones aren't, leading to inconsitencies (for
example, if you change req.filename, req.canonical_filename isn't
changed). Should we try to solve this and provide clear definition of
the various parts of a request for mod_python 3.3 ?
That would make sense. I'm wondering how often people make use of
req.canonical_filename (CFN*)? Also, just how would the CFN be
adjusted if the user code changes req.filename, since the user is free
to put any string in there they want? Maybe CFN just gets changed to the
same string. Hopefully Graham will shed some light on this, since it was
his use case.
Regards,
Jim
* Because I can't type canonical_filename the same way twice. Stupid
fingers.
Re: Various musings about the request URL / URI / whatever
Daniel J. Popowich wrote: Jim Gallacher writes: Nicolas Lehuen wrote: Second question, if there isn't any simpler way to do this, should we add it to mod_python ? Either as a function like above in mod_python.util, or as a member of the request object (named something like url to match the other member named uri, but that's just teasing). I'm not against it, but for my purposes I think it would be more useful for parsed_uri to just work properly. Here, here!! I've wanted parsed_uri to work as expected for quite some time...I'm actually in a position where I could devote some time to tracking this down. If apache doesn't provide it, I think mod_python should at least fill it in, right? +1 Can someone knudge me in the right direction to start? Haven't poked around apache source and/or developer docs in years. All I can say is grep is your friend. :) I've found http://docx.webperf.org to be useful. Unfortunately you can only drill down into the header files, not c files (unless I'm missing something). I might even be tempted to generate my own local copy of the apache docs using doxygen so that the c-files get included. I've been playing with doxygen + mod_python and it's pretty cool. Searching docx for parse_uri turns up ap_parse_uri. http://docx.webperf.org/group__APACHE__CORE__PROTO.html#ga44 Grab the src and put grep to work. I'll dig in and help any way I can. Jim
Re: Various musings about the request URL / URI / whatever
Gregory (Grisha) Trubetskoy wrote: On Tue, 29 Nov 2005, Jim Gallacher wrote: Daniel J. Popowich wrote: Here, here!! I've wanted parsed_uri to work as expected for quite some time...I'm actually in a position where I could devote some time to tracking this down. If apache doesn't provide it, I think mod_python should at least fill it in, right? +1 I don't know what the specific issue is with parsed_uri, if this is a mod_python bug it should just be fixed BUT if this is an issue with httpd, I don't think we should cover the problem up by having mod_python fix it. Since we are part of the HTTP Server project, we should just fix it in httpd. Either way, it should be fixed. In case anyone is not familiar with the issue, a request for http://example.com/tests/mptest?view=form currently gives a tuple that looks something like this: (None, None, None, None, None, None, '/tests/mptest', 'view=form', None) which is not what we expect. This is what the mod_python docs have to say: parsed_uri Tuple. The URI broken down into pieces. (scheme, hostinfo, user, password, hostname, port, path, query, fragment). The apache module defines a set of URI_* constants that should be used to access elements of this tuple. Example: fname = req.parsed_uri[apache.URI_PATH] (Read-Only) Jim
Re: [jira] Updated: (MODPYTHON-78) No support for Apache 2.2 yet
Nicolas Lehuen (JIRA) wrote: [ http://issues.apache.org/jira/browse/MODPYTHON-78?page=all ] Nicolas Lehuen updated MODPYTHON-78: Summary: No support for Apache 2.2 yet (was: No support for Apache 2.1 yet) Now that Apache 2.2 is out, and mod_python 3.2 close to release, maybe it's time to have a look at this for mod_python 3.3 ? +1
Re: Apache 2.2 released - apparently breaks the mod API compatibility with 2.0
Nicolas Lehuen wrote: I'll have to wait for the Win32 source code tree to be released to build it and test your patch. Hopefully it'll be out soon. Is there a wait to use macro directives so that we don't need to maintain two separate branches ? A define that we could pass when building mod_python to select the Apache version we're building against, maybe ? I'm sure there must be. If not perhaps we roll our own using some configure option, eg --with-apache2.2. I think it would be a big mistake to try and maintain 2 branches for apache 2.0 and 2.2. They can't be *that* different (and judging by Jorey's patch they are not). Jim
Re: Does 3.2 still support python 2.2?
I figured we had moved on to 2.3, but I wanted to make sure I wasn't missing something before I changed the docs. I'm not sure if there was a formal decision on this or everyone just assumed it was true. Perhaps a pronoucement from Grisha to make it offical? If python 2.2 support has been dropped then it needs needs to be mentioned in changes section of the docs, and the README as well. Jim Nick wrote: I'm pretty sure we've had a few discussions about being able to use certain functions and modules because they became available in 2.3, and that's what mod_python was going to require. Like the bsddb database version for your session code, for example. Nick Jim Gallacher wrote: From the 3.2.5b doc: (http://www.modpython.org/live/mod_python-3.2.5b/doc-html/inst-prerequisites.html) 2.1 Prerequisites * Python 2.2.1 or later. Earlier versions of Python will not work. Is this still true or have we dropped support for python 2.3? Has anybody tested using python 2.2.1? Jim
Re: Does 3.2 still support python 2.2?
OK then, good enough. Next question. Should we bump the Apache version requirement as well. Currently the docs state that Apache 2.0.40 or later is required. I don't recall seeing anyone testing mod_python 3.2 on anything less than apache 2.0.53. I don't know if there are any changes between 40 and 53 that may have a negative impact, but if we haven't actually tested on the earlier versions are we just asking for trouble? Jim Nick wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 http://article.gmane.org/gmane.comp.apache.mod-python.devel/865 Jim Gallacher wrote: | I figured we had moved on to 2.3, but I wanted to make sure I wasn't | missing something before I changed the docs. I'm not sure if there was a | formal decision on this or everyone just assumed it was true. Perhaps a | pronoucement from Grisha to make it offical? | | If python 2.2 support has been dropped then it needs needs to be | mentioned in changes section of the docs, and the README as well. | | Jim | | Nick wrote: | | I'm pretty sure we've had a few discussions about being able to use | certain functions and modules because they became available in 2.3, | and that's what mod_python was going to require. Like the bsddb | database version for your session code, for example. | | Nick | | Jim Gallacher wrote: | | From the 3.2.5b doc: | | (http://www.modpython.org/live/mod_python-3.2.5b/doc-html/inst-prerequisites.html) | | | 2.1 Prerequisites | | * Python 2.2.1 or later. Earlier versions of Python will not work. | | | Is this still true or have we dropped support for python 2.3? Has | anybody tested using python 2.2.1? | | Jim | | | | -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFDmj4Cv4zJ7LQ+i84RAh9VAJ9rWpumf/Bdky9NuK0bvX96NHrmQQCeKSDD JAF18Qqe3CvDezgOww9599A= =tlHN -END PGP SIGNATURE-
3.2.6b?
We had talked about doing a 3.2 final release just after ApacheCon. A couple of things have cropped up which we have (or should) fix, but these will not be substantial changes from 3.2.5b. As such I think we should do another beta followed very quickly by a final release. Any new bugs that crop up will have to wait for a 3.2 bug fix release sometime in January or February. How about a beta on Dec 18 (if possible) followed by a final release before Dec 31? Jim
Re: [jira] Created: (MODPYTHON-98) wrong handler supplied to req.add_handler() generates error
My eyes keep glazing over every time I read through MODPYTHON-98. This is not a reflection on Graham's excellent writing. I should probably just drink some more coffee. :) As Graham suggests I think raising an exception is the right way to go, along with the documentation change to reflect this. Obviously we should fix the segfault as well. Jim
Re: [jira] Commented: (MODPYTHON-99) accessing some request or server object members causes a segfault
I declare the voting over and empty tuple wins in a landslide. :) I'll commit the changes shortly. Jim Nicolas Lehuen wrote: +1 for the empty tuple too. Regards, Nicolas 2005/12/17, Jim Gallacher [EMAIL PROTECTED]: Nicolas Lehuen (JIRA) wrote: [ http://issues.apache.org/jira/browse/MODPYTHON-99?page=comments#action_12360042 ] Nicolas Lehuen commented on MODPYTHON-99: - OK, if we modify tuple_from_array_header and tuple_from_method_list, here are the members that would be defined as an empty tuple rather than a None value when the underlying Apache data is not defined : server.names server.wildnames req.allowed_xmethods req.allowed_methods req.content_languages I don't know if we should change the behaviour of those members or simply alter the unit tests, though. Let's make a decision on this, get it fixed, and move 3.2 forward. +1 for returning an empty tuple. It seems more logical to me it was already broken so making the change shouldn't affect anyone's code. Jim
Re: [jira] Commented: (MODPYTHON-98) wrong handler supplied to req.add_handler() generates error
Jim Gallacher (JIRA) wrote: [ http://issues.apache.org/jira/browse/MODPYTHON-98?page=comments#action_12362399 ] Jim Gallacher commented on MODPYTHON-98: Applied Graham's suggestions so all these related issues can be considered fixed. Still need to write some unit tests before this issue is marked as resolved. Graham, I've committed the following unit tests for MODPYTHON-98: test_req_add_bad_handler test_req_add_empty_handler_string test_accesshandler_add_handler_to_empty_hl Can you take a look and let me know if you think these properly cover the issue? If they look ok I'll roll the 3.2.6 beta. Jim
Re: [jira] Commented: (MODPYTHON-98) wrong handler supplied to req.add_handler() generates error
Graham Dumpleton wrote: On 12/01/2006, at 11:10 AM, Jim Gallacher wrote: Jim Gallacher (JIRA) wrote: [ http://issues.apache.org/jira/browse/MODPYTHON-98? page=comments#action_12362399 ] Jim Gallacher commented on MODPYTHON-98: Applied Graham's suggestions so all these related issues can be considered fixed. Still need to write some unit tests before this issue is marked as resolved. Graham, I've committed the following unit tests for MODPYTHON-98: test_req_add_bad_handler test_req_add_empty_handler_string test_accesshandler_add_handler_to_empty_hl Can you take a look and let me know if you think these properly cover the issue? If they look ok I'll roll the 3.2.6 beta. I get a failure on test_req_add_empty_handler_string. I don't know if this is because of other local hacks I have in place or not. But for it to pass I need the following change to the test.py: This test was of some concern, as I thought the error log message might be inconsistent. Index: test.py === --- test.py (revision 368329) +++ test.py (working copy) @@ -516,8 +516,8 @@ f = open(os.path.join(SERVER_ROOT, logs/error_log)) log = f.read() f.close() -if log.find(contains no 'handler') == -1: -self.fail(Could not find contains no 'handler' in error_log) +if log.find(No module named) == -1: +self.fail(Could not find No module named in error_log) def test_accesshandler_add_handler_to_empty_hl_conf(self): # Note that there is no PythonHandler specified in the the VirtualHost The messages in the error log being: [Thu Jan 12 21:39:35 2006] [notice] mod_python: (Re)importing module '' [Thu Jan 12 21:39:35 2006] [error] [client 127.0.0.1] PythonHandler : Traceback (most recent call last): [Thu Jan 12 21:39:35 2006] [error] [client 127.0.0.1] PythonHandler : File /System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/mod_python/apache.py, line 284, in HandlerDispatch\nlog=debug) [Thu Jan 12 21:39:35 2006] [error] [client 127.0.0.1] PythonHandler : File /System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/mod_python/apache.py, line 468, in import_module\nf, p, d = imp.find_module(parts[i], path) [Thu Jan 12 21:39:35 2006] [error] [client 127.0.0.1] PythonHandler : ImportError: No module named Your error message actually makes more sense, as the last line in my error_log for this exception mentions PIL. I have *no* idea where that comes from but was hoping that the final line would at least always contain the string contains no 'handler'. Here is my error_log: [Thu Jan 12 08:25:13 2006] [error] [client 127.0.0.1] req_add_empty_handler_string [Thu Jan 12 08:25:13 2006] [notice] mod_python: (Re)importing module '' [Thu Jan 12 08:25:13 2006] [error] [client 127.0.0.1] PythonHandler : Traceback (most recent call last): [Thu Jan 12 08:25:13 2006] [error] [client 127.0.0.1] PythonHandler : File /usr/lib/python2.3/site-packages/mod_python/apache.py, line 291, in HandlerDispatch\narg=req, silent=hlist.silent) [Thu Jan 12 08:25:13 2006] [error] [client 127.0.0.1] PythonHandler : File /usr/lib/python2.3/site-packages/mod_python/apache.py, line 531, in resolve_object\nraise AttributeError, s [Thu Jan 12 08:25:13 2006] [error] [client 127.0.0.1] PythonHandler : AttributeError: module '/usr/lib/python2.3/site-packages/PIL/__init__.pyc' contains no 'handler' Perhaps some other people could check out trunk, run the tests and let me know what shows up in the error_log after the req_add_empty_hander_string line. Thanks, Jim
Re: [jira] Commented: (MODPYTHON-98) wrong handler supplied to req.add_handler() generates error
Ok, this is weird. I've run the tests on a couple of my qemu images and I get the same output as Nicolas and Graham in the error log, but a different output for my development machine. I think I'll rewrite the test to just check the response from the request rather than rooting around in the error_log. The point of the test is to make sure that an exception is raised when req.add_handler(PythonHandler, ) is attempted, so there a couple of different ways to go. I'll make the change and then roll the new beta tarball. Jim Nicolas Lehuen wrote: Hi all, I'm back after a few days without Internet access... I also have the same test failure as Graham. Here is my error log : [Thu Jan 12 20:11:24 2006] [error] [client 127.0.0.1] req_add_empty_handler_string [Thu Jan 12 20:11:24 2006] [notice] mod_python: (Re)importing module '' [Thu Jan 12 20:11:24 2006] [error] [client 127.0.0.1] PythonHandler : Traceback (most recent call last): [Thu Jan 12 20:11:24 2006] [error] [client 127.0.0.1] PythonHandler : File E:\\Python24\\Lib\\site-packages\\mod_python\\apache.py, line 287, in HandlerDispatch\nlog=debug) [Thu Jan 12 20:11:24 2006] [error] [client 127.0.0.1] PythonHandler : File E:\\Python24\\Lib\\site-packages\\mod_python\\apache.py, line 461, in import_module\nf, p, d = imp.find_module(parts[i], path) [Thu Jan 12 20:11:24 2006] [error] [client 127.0.0.1] PythonHandler : ImportError: No module named [Thu Jan 12 20:11:25 2006] [notice] mod_python: (Re)importing module 'tests' [Thu Jan 12 20:11:25 2006] [error] [client 127.0.0.1] accesshandler_add_handler_to_empty_hl Regards, Nicolas 2006/1/12, Jim Gallacher [EMAIL PROTECTED]: Graham Dumpleton wrote: On 12/01/2006, at 11:10 AM, Jim Gallacher wrote: Jim Gallacher (JIRA) wrote: [ http://issues.apache.org/jira/browse/MODPYTHON-98? page=comments#action_12362399 ] Jim Gallacher commented on MODPYTHON-98: Applied Graham's suggestions so all these related issues can be considered fixed. Still need to write some unit tests before this issue is marked as resolved. Graham, I've committed the following unit tests for MODPYTHON-98: test_req_add_bad_handler test_req_add_empty_handler_string test_accesshandler_add_handler_to_empty_hl Can you take a look and let me know if you think these properly cover the issue? If they look ok I'll roll the 3.2.6 beta. I get a failure on test_req_add_empty_handler_string. I don't know if this is because of other local hacks I have in place or not. But for it to pass I need the following change to the test.py: This test was of some concern, as I thought the error log message might be inconsistent. Index: test.py === --- test.py (revision 368329) +++ test.py (working copy) @@ -516,8 +516,8 @@ f = open(os.path.join(SERVER_ROOT, logs/error_log)) log = f.read() f.close() -if log.find(contains no 'handler') == -1: -self.fail(Could not find contains no 'handler' in error_log) +if log.find(No module named) == -1: +self.fail(Could not find No module named in error_log) def test_accesshandler_add_handler_to_empty_hl_conf(self): # Note that there is no PythonHandler specified in the the VirtualHost The messages in the error log being: [Thu Jan 12 21:39:35 2006] [notice] mod_python: (Re)importing module '' [Thu Jan 12 21:39:35 2006] [error] [client 127.0.0.1] PythonHandler : Traceback (most recent call last): [Thu Jan 12 21:39:35 2006] [error] [client 127.0.0.1] PythonHandler : File /System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/mod_python/apache.py, line 284, in HandlerDispatch\nlog=debug) [Thu Jan 12 21:39:35 2006] [error] [client 127.0.0.1] PythonHandler : File /System/Library/Frameworks/Python.framework/Versions/2.3/lib/ python2.3/site-packages/mod_python/apache.py, line 468, in import_module\nf, p, d = imp.find_module(parts[i], path) [Thu Jan 12 21:39:35 2006] [error] [client 127.0.0.1] PythonHandler : ImportError: No module named Your error message actually makes more sense, as the last line in my error_log for this exception mentions PIL. I have *no* idea where that comes from but was hoping that the final line would at least always contain the string contains no 'handler'. Here is my error_log: [Thu Jan 12 08:25:13 2006] [error] [client 127.0.0.1] req_add_empty_handler_string [Thu Jan 12 08:25:13 2006] [notice] mod_python: (Re)importing module '' [Thu Jan 12 08:25:13 2006] [error] [client 127.0.0.1] PythonHandler : Traceback (most recent call last): [Thu Jan 12 08:25:13 2006] [error] [client 127.0.0.1] PythonHandler : File /usr/lib/python2.3/site-packages/mod_python/apache.py, line 291, in HandlerDispatch\narg=req, silent=hlist.silent) [Thu Jan 12 08:25:13 2006] [error] [client 127.0.0.1] PythonHandler : File /usr/lib/python2.3/site-packages
Re: [jira] Commented: (MODPYTHON-98) wrong handler supplied to req.add_handler()generateserror
Graham Dumpleton wrote:
Jim Gallacher wrote ..
It's a strange one. When I move site-packages/PIL to
site-packages/PIL.bak (leaving PIL.pth as is) and run the tests I get
the same output as Graham and Nicolas. I'm just going to ignore this for
the time being and go with a refactored unit test.
I am making a guess that it is because of the following:
~ [510]$ touch .py
~ [511]$ python
Python 2.3 (#1, Sep 13 2003, 00:49:11)
[GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin
Type help, copyright, credits or license for more information.
import imp
imp.find_module('',None)
(open file '.py', mode 'U' at 0x602e0, '.py', ('.py', 'U', 1))
Ie., look for a .py file in PIL.
There is no .py in site-packages/PIL/ but it does look like the problem
is PIL related.
import imp
imp.find_module('',None)
(None, '/usr/lib/python2.3/site-packages/PIL/', ('', '', 5))
I've changed the unit test to avoid the problem so I don't think it's
worth wasting any more time.
Jim
mod_python 3.2.6b available for testing
A new mod_python 3.2.6 beta tarball is now available for testing. Nicolas has built windows versions for Python 2.4 and Python 2.3 which should also be available at www.modpython.org/dist shortly. This release is similar to 3.2.5b but fixes a couple of issues - MODPYTHON-95, 96, 97, 98, 99, 105, 106. I think if we get enough +1 votes for this version on python-dev that we should jump right to a 3.2.6 final release rather than go to a wider beta release. So far we have: +1 Linux Debian sid, apache 2.0.55 (mpm-prefork), python 2.3 +1 Linux Debian sarge, apache 2.0.54 (mpm-worker), python 2.3 +1 Windows XP, python 2.4 +1 Windows 2000, python 2.3 +1 Mac OS X 10.3, Apache 2.0.55, python 2.3 Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_ list*! (Not the [EMAIL PROTECTED] list, and preferably not me personally). The files are (temporarily) available here: http://www.modpython.org/dist/ Please download it, then do the usual $ ./configure --with-apxs=/wherever/it/is $ make $ (su) # make install Then (as non-root user!) $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the details (the versions of OS, Python and Apache, the test output, and suggestions, if any). Thank you, Jim Ga
mod_python 3.2.6 (Final!) available for testing
Good news everyone! I made a mistake in tagging 3.2.6 as beta instead of final. The new tarball is now available for testing. This is the same code as yesterday's 3.2.6b.tgz but with the correct version information. This is the one we've all been waiting for! :) Here are the rules: In order for a file to be officially announced, it has to be tested by developers on the dev list. Anyone subscribed to this list can (and should feel obligated to :-) ) test it, and provide feedback *to _this_ list*! (Not the [EMAIL PROTECTED] list, and preferably not me personally). The files are (temporarily) available here: http://www.modpython.org/dist/ Please download it, then do the usual $ ./configure --with-apxs=/wherever/it/is $ make $ (su) # make install Then (as non-root user!) $ cd test $ python test.py And see if any tests fail. If they pass, send a +1 to the list, if they fail, send the details (the versions of OS, Python and Apache, the test output, and suggestions, if any). Thank you, Jim Ga
Re: mod_python 3.2.6b available for testing
Michel Jouvin wrote: 0 : HP Tru64, mpm-worker In fact 3.2.6b runs as 3.2.5b. Basically in real context it works except the fact that after a segfault in the Apache child or a signal received other than TERM or USR1, it doesn't reinitialize properly and forbid proper reinitialization of the new Apache child (I already sent mails about that, may be I need to fill it in in the bug database ?). Hi Michel, I went back and looked at the previous thread and it was left hanging. Not having access to a Tru64 machine and being unable to reproduce the problem on another platform does present a challenge. The differences between 3.2.5b and 3.2.6 are minor so it's not a suprise that you are still seeing the same problem. The good news is that a potential segfault condition was fixed so at least you are slightly less likely to exercise your particular bug. It would be best to create a new JIRA issue so that the bug doesn't get lost in the mailing list again. Log your issue at http://issues.apache.org/jira/browse/MODPYTHON. Thanks, Jim
