Re: 3.2.6 test period - how long do we wait?
I'd give a conditional +1 - I think it'd be good to sort out why previous releases passed on FreeBSD and this one does not. Perhaps people who ran FreeBSD tests could try an earlier mod_python version test to see if it too fails? Grisha On Tue, 31 Jan 2006, Jim Gallacher wrote: Good enough for me. Shall we vote? If so I am: +1 Jim Graham Dumpleton wrote: I am having problems with posts to python-dev mailing list from home occassionally disappearing in a black hole. Thus my post on this topic before Jim brought it up in the first place vanished. What I has said was: this code runs smoothly, i.e. no segfaults, all tests passed: FreeBSD 4.9: Apache/2.0.50 (prefork) Python/2.3.4 Apache/2.0.55 (prefork) Python/2.4.2 Okay, this is good. If there is a general consensus that this is a reasonable solution, the question is what should be done with 3.2.6. Should it be released with this problem in it and fix it later in 3.3? Personally, I really don't think that connection handlers in mod_python get used by anyone and so don't see a pressing need to go fixing it right now. Thus I would say go with 3.2.6 as final with how it is. Is there anything else of concern that is holding us up now on an official release of 3.2.6? In other words, I agree with Nicolas, we should just release it. If someone actually came up and said they were using connection handlers for something significant, then I might change this view, but I doubt anyone is using them. I would also point out that JIRA lists lots of other issues, some of them minor and trivial to fix but which would be more worthwhile than fixing than this connection handler issue. We have already said we will defer these in the interest of getting a final version of 3.2 out sooner rather than later. Thus, as much as I would prefer to see as much as possible fixed, I think we just have defer further changes to next version. Graham Jim Gallacher wrote .. Nicolas Lehuen wrote: OK, so shall we schedule the 3.2.x release for 2007, then ? As for the Apache 2.2 version, what if we roll in your suggested patch, Jim, then discover a bunch of problem related to it during the beta tests ? Will we wait until they are all fixed to release the 3.2 version ? Apache 2.2 is quite new so we'll likely to have to squish bugs, due for example to new interaction between Apache filters and mod_python. That's a wild guess but filters have been modified in Apache 2.2 so I'm sure something evil lurks there. I'm totally happy to defer changes for Apache 2.2 as I don't personally plan on migrating any time soon. The only reason I brought it up is that some of the code which seems to be causing issues for Apache 2.2 happens to reside in connobject.c, which is an area of interest for the _conn_read issue. Even if we were to roll fixes for apache 2.2 into mp 3.2, I'm not suggesting we advertise apache 2.2 compatibility just yet. Or we could simply forget about making the release one day and tell every user to use the latest snapshot from subversion. Sorry to be like that, but we have users out there that would be perfectly happy with the current state of the 3.2.6 version, and a lot of our answers on the mailing list are "yup, we know this bug, it's already been fixed one year ago, but don't worry, you'll get the bugfix soon enough". Point taken, and it is a very good point indeed. Once again, it seems that no regression have been introduced in 3.2.6 vs 3.1.4, so we should release it ASAP and try to keep a steady release rythm afterwards. When we'll get momentum we'll solve a bunch of problem pretty fast, but it's been a year now that we are paralysed by perfectionism. What could be worse than leaving our users out there with the current 3.1.4 version ? I *still* wonder why the whole ConnectionHandler issue was not seen on FreeBSD with mp 3.1.4 though. I'll need to check the svn logs again but I'm pretty sure this is not a new unit test. That being said I'm not personally concerned about this issue since it doesn't affect every platform (or more selfishly I should say it doesn't affect the OS I'm working on) and I suspect that not many people are directly interacting with the connection object. What's that old line about open source software... Release early, release often? Jim 2006/1/31, Jim Gallacher <[EMAIL PROTECTED]>: I assume we will be doing a 3.2.7 release if Graham's fix for the ConnectionHandler / MODPYTHON-102 problem works? If that is the case I wonder if we should roll in the changes to support apache 2.2. I scanned mod_python for deprecated or removed apr calls and can find only one (apr_sockaddr_port_get), plus the missing APR_STATUS_IS_SUCCESS macro. The original macro is: #define APR_STATUS_IS_SUCCESS(s) ((s) == APR_SUCCESS \ || (s) == APR_OS_START_SYSERR + NO_ERROR) The discussion on httpd-dev suggested that this macro should be substituted with a simple test such as "if (rc != APR_
Re: 3.2.6 test period - how long do we wait?
+1 Release what's fixed already, and then keep going afterwards. But can somebody address MODPYTHON-53 please. (updating modpython.org website). -- Deron Meranda
Re: 3.2.6 test period - how long do we wait?
Good enough for me. Shall we vote? If so I am: +1 Jim Graham Dumpleton wrote: I am having problems with posts to python-dev mailing list from home occassionally disappearing in a black hole. Thus my post on this topic before Jim brought it up in the first place vanished. What I has said was: this code runs smoothly, i.e. no segfaults, all tests passed: FreeBSD 4.9: Apache/2.0.50 (prefork) Python/2.3.4 Apache/2.0.55 (prefork) Python/2.4.2 Okay, this is good. If there is a general consensus that this is a reasonable solution, the question is what should be done with 3.2.6. Should it be released with this problem in it and fix it later in 3.3? Personally, I really don't think that connection handlers in mod_python get used by anyone and so don't see a pressing need to go fixing it right now. Thus I would say go with 3.2.6 as final with how it is. Is there anything else of concern that is holding us up now on an official release of 3.2.6? In other words, I agree with Nicolas, we should just release it. If someone actually came up and said they were using connection handlers for something significant, then I might change this view, but I doubt anyone is using them. I would also point out that JIRA lists lots of other issues, some of them minor and trivial to fix but which would be more worthwhile than fixing than this connection handler issue. We have already said we will defer these in the interest of getting a final version of 3.2 out sooner rather than later. Thus, as much as I would prefer to see as much as possible fixed, I think we just have defer further changes to next version. Graham Jim Gallacher wrote .. Nicolas Lehuen wrote: OK, so shall we schedule the 3.2.x release for 2007, then ? As for the Apache 2.2 version, what if we roll in your suggested patch, Jim, then discover a bunch of problem related to it during the beta tests ? Will we wait until they are all fixed to release the 3.2 version ? Apache 2.2 is quite new so we'll likely to have to squish bugs, due for example to new interaction between Apache filters and mod_python. That's a wild guess but filters have been modified in Apache 2.2 so I'm sure something evil lurks there. I'm totally happy to defer changes for Apache 2.2 as I don't personally plan on migrating any time soon. The only reason I brought it up is that some of the code which seems to be causing issues for Apache 2.2 happens to reside in connobject.c, which is an area of interest for the _conn_read issue. Even if we were to roll fixes for apache 2.2 into mp 3.2, I'm not suggesting we advertise apache 2.2 compatibility just yet. Or we could simply forget about making the release one day and tell every user to use the latest snapshot from subversion. Sorry to be like that, but we have users out there that would be perfectly happy with the current state of the 3.2.6 version, and a lot of our answers on the mailing list are "yup, we know this bug, it's already been fixed one year ago, but don't worry, you'll get the bugfix soon enough". Point taken, and it is a very good point indeed. Once again, it seems that no regression have been introduced in 3.2.6 vs 3.1.4, so we should release it ASAP and try to keep a steady release rythm afterwards. When we'll get momentum we'll solve a bunch of problem pretty fast, but it's been a year now that we are paralysed by perfectionism. What could be worse than leaving our users out there with the current 3.1.4 version ? I *still* wonder why the whole ConnectionHandler issue was not seen on FreeBSD with mp 3.1.4 though. I'll need to check the svn logs again but I'm pretty sure this is not a new unit test. That being said I'm not personally concerned about this issue since it doesn't affect every platform (or more selfishly I should say it doesn't affect the OS I'm working on) and I suspect that not many people are directly interacting with the connection object. What's that old line about open source software... Release early, release often? Jim 2006/1/31, Jim Gallacher <[EMAIL PROTECTED]>: I assume we will be doing a 3.2.7 release if Graham's fix for the ConnectionHandler / MODPYTHON-102 problem works? If that is the case I wonder if we should roll in the changes to support apache 2.2. I scanned mod_python for deprecated or removed apr calls and can find only one (apr_sockaddr_port_get), plus the missing APR_STATUS_IS_SUCCESS macro. The original macro is: #define APR_STATUS_IS_SUCCESS(s) ((s) == APR_SUCCESS \ || (s) == APR_OS_START_SYSERR + NO_ERROR) The discussion on httpd-dev suggested that this macro should be substituted with a simple test such as "if (rc != APR_SUCCESS)", and the '||' condition was not likely used. So that we are making the fewest possible changes to our current 3.2 codebase, I'd suggest reimplenting APR_STATUS_IS_SUCCESS in our code, and then removing it 3.3. This will give us lot's of time as we work on 3.3 to discover if the
Re: 3.2.6 test period - how long do we wait?
I am having problems with posts to python-dev mailing list from home occassionally disappearing in a black hole. Thus my post on this topic before Jim brought it up in the first place vanished. What I has said was: >> this code runs smoothly, i.e. no segfaults, all tests passed: >> FreeBSD 4.9: >> >> Apache/2.0.50 (prefork) Python/2.3.4 >> Apache/2.0.55 (prefork) Python/2.4.2 > > Okay, this is good. > > If there is a general consensus that this is a reasonable solution, > the question is what should be done with 3.2.6. Should it be released > with this problem in it and fix it later in 3.3? > > Personally, I really don't think that connection handlers in > mod_python get used by anyone and so don't see a pressing need to go > fixing it right now. Thus I would say go with 3.2.6 as final with how > it is. > > Is there anything else of concern that is holding us up now on an > official release of 3.2.6? In other words, I agree with Nicolas, we should just release it. If someone actually came up and said they were using connection handlers for something significant, then I might change this view, but I doubt anyone is using them. I would also point out that JIRA lists lots of other issues, some of them minor and trivial to fix but which would be more worthwhile than fixing than this connection handler issue. We have already said we will defer these in the interest of getting a final version of 3.2 out sooner rather than later. Thus, as much as I would prefer to see as much as possible fixed, I think we just have defer further changes to next version. Graham Jim Gallacher wrote .. > Nicolas Lehuen wrote: > > OK, so shall we schedule the 3.2.x release for 2007, then ? > > > > As for the Apache 2.2 version, what if we roll in your suggested > > patch, Jim, then discover a bunch of problem related to it during the > > beta tests ? Will we wait until they are all fixed to release the 3.2 > > version ? Apache 2.2 is quite new so we'll likely to have to squish > > bugs, due for example to new interaction between Apache filters and > > mod_python. That's a wild guess but filters have been modified in > > Apache 2.2 so I'm sure something evil lurks there. > > I'm totally happy to defer changes for Apache 2.2 as I don't personally > plan on migrating any time soon. The only reason I brought it up is that > some of the code which seems to be causing issues for Apache 2.2 happens > to reside in connobject.c, which is an area of interest for the > _conn_read issue. Even if we were to roll fixes for apache 2.2 into mp > 3.2, I'm not suggesting we advertise apache 2.2 compatibility just yet. > > > > Or we could simply forget about making the release one day and > > tell every user to use the latest snapshot from subversion. Sorry to > > be like that, but we have users out there that would be perfectly > > happy with the current state of the 3.2.6 version, and a lot of our > > answers on the mailing list are "yup, we know this bug, it's already > > been fixed one year ago, but don't worry, you'll get the bugfix soon > > enough". > > Point taken, and it is a very good point indeed. > > > Once again, it seems that no regression have been introduced in 3.2.6 > > vs 3.1.4, so we should release it ASAP and try to keep a steady > > release rythm afterwards. When we'll get momentum we'll solve a bunch > > of problem pretty fast, but it's been a year now that we are paralysed > > by perfectionism. What could be worse than leaving our users out there > > with the current 3.1.4 version ? > > I *still* wonder why the whole ConnectionHandler issue was not seen on > FreeBSD with mp 3.1.4 though. I'll need to check the svn logs again but > I'm pretty sure this is not a new unit test. > > That being said I'm not personally concerned about this issue since it > doesn't affect every platform (or more selfishly I should say it doesn't > affect the OS I'm working on) and I suspect that not many people are > directly interacting with the connection object. > > What's that old line about open source software... Release early, > release often? > > Jim > > > 2006/1/31, Jim Gallacher <[EMAIL PROTECTED]>: > > > >>I assume we will be doing a 3.2.7 release if Graham's fix for the > >>ConnectionHandler / MODPYTHON-102 problem works? > >> > >>If that is the case I wonder if we should roll in the changes to support > >>apache 2.2. I scanned mod_python for deprecated or removed apr calls > and > >>can find only one (apr_sockaddr_port_get), plus the missing > >>APR_STATUS_IS_SUCCESS macro. > >> > >>The original macro is: > >> > >>#define APR_STATUS_IS_SUCCESS(s) ((s) == APR_SUCCESS \ > >> || (s) == APR_OS_START_SYSERR + NO_ERROR) > >> > >> > >>The discussion on httpd-dev suggested that this macro should be > >>substituted with a simple test such as "if (rc != APR_SUCCESS)", and > the > >>'||' condition was not likely used. So that we are making the fewest > >>possible changes to our current 3.2 codebase
Re: 3.2.6 test period - how long do we wait?
Deron Meranda wrote: On 1/31/06, Jim Gallacher <[EMAIL PROTECTED]> wrote: apache 2.2. I scanned mod_python for deprecated or removed apr calls and can find only one (apr_sockaddr_port_get), plus the missing APR_STATUS_IS_SUCCESS macro. The apr_sockaddr_port_get() call was introduced by me to support IPv6 in modpython with MODPYTHON-64. Looking at the APR 1.0 CHANGES file (accompaning httpd 2.2): *) The following deprecated interfaces have been removed: apr_sockaddr_port_get-> (access directly) I'm trying to review the APR source to see exactly what that means and the conditions under which it's safe. We don't want to accidentally pull the wrong field and have endianness problems. BTW, I want to get this one fixed before release rather than reverting to the pre-MODPYTHON-64 code...or we'll break under IPv6 again. I say we leave it as is (ie continue use apr_sockaddr_port_get) and worry about it when we do some sort of comprehensive review for apache 2.2 compatibility. Jim
Re: 3.2.6 test period - how long do we wait?
Nicolas Lehuen wrote: OK, so shall we schedule the 3.2.x release for 2007, then ? As for the Apache 2.2 version, what if we roll in your suggested patch, Jim, then discover a bunch of problem related to it during the beta tests ? Will we wait until they are all fixed to release the 3.2 version ? Apache 2.2 is quite new so we'll likely to have to squish bugs, due for example to new interaction between Apache filters and mod_python. That's a wild guess but filters have been modified in Apache 2.2 so I'm sure something evil lurks there. I'm totally happy to defer changes for Apache 2.2 as I don't personally plan on migrating any time soon. The only reason I brought it up is that some of the code which seems to be causing issues for Apache 2.2 happens to reside in connobject.c, which is an area of interest for the _conn_read issue. Even if we were to roll fixes for apache 2.2 into mp 3.2, I'm not suggesting we advertise apache 2.2 compatibility just yet. Or we could simply forget about making the release one day and tell every user to use the latest snapshot from subversion. Sorry to be like that, but we have users out there that would be perfectly happy with the current state of the 3.2.6 version, and a lot of our answers on the mailing list are "yup, we know this bug, it's already been fixed one year ago, but don't worry, you'll get the bugfix soon enough". Point taken, and it is a very good point indeed. Once again, it seems that no regression have been introduced in 3.2.6 vs 3.1.4, so we should release it ASAP and try to keep a steady release rythm afterwards. When we'll get momentum we'll solve a bunch of problem pretty fast, but it's been a year now that we are paralysed by perfectionism. What could be worse than leaving our users out there with the current 3.1.4 version ? I *still* wonder why the whole ConnectionHandler issue was not seen on FreeBSD with mp 3.1.4 though. I'll need to check the svn logs again but I'm pretty sure this is not a new unit test. That being said I'm not personally concerned about this issue since it doesn't affect every platform (or more selfishly I should say it doesn't affect the OS I'm working on) and I suspect that not many people are directly interacting with the connection object. What's that old line about open source software... Release early, release often? Jim 2006/1/31, Jim Gallacher <[EMAIL PROTECTED]>: I assume we will be doing a 3.2.7 release if Graham's fix for the ConnectionHandler / MODPYTHON-102 problem works? If that is the case I wonder if we should roll in the changes to support apache 2.2. I scanned mod_python for deprecated or removed apr calls and can find only one (apr_sockaddr_port_get), plus the missing APR_STATUS_IS_SUCCESS macro. The original macro is: #define APR_STATUS_IS_SUCCESS(s) ((s) == APR_SUCCESS \ || (s) == APR_OS_START_SYSERR + NO_ERROR) The discussion on httpd-dev suggested that this macro should be substituted with a simple test such as "if (rc != APR_SUCCESS)", and the '||' condition was not likely used. So that we are making the fewest possible changes to our current 3.2 codebase, I'd suggest reimplenting APR_STATUS_IS_SUCCESS in our code, and then removing it 3.3. This will give us lot's of time as we work on 3.3 to discover if there are any problems droping the APR_OS_START_SYSERR part of the test. Jim
Re: 3.2.6 test period - how long do we wait?
On Tue, 31 Jan 2006, Jim Gallacher wrote: I assume we will be doing a 3.2.7 release if Graham's fix for the ConnectionHandler / MODPYTHON-102 problem works? Unfortunately, the answer is yes... As much as we'd like to release it sooner, I think it's important to not loose the perspective that quality over speed is what we favor. In other words - if it takes a little longer to get the quality we want, then that's just fine. :-) Grisha
Re: 3.2.6 test period - how long do we wait?
On 1/31/06, Deron Meranda <[EMAIL PROTECTED]> wrote: > On 1/31/06, Jim Gallacher <[EMAIL PROTECTED]> wrote: > > apache 2.2. I scanned mod_python for deprecated or removed apr calls and > > can find only one (apr_sockaddr_port_get), plus the missing > > APR_STATUS_IS_SUCCESS macro. > > The apr_sockaddr_port_get() call was introduced by me to support > IPv6 in modpython with MODPYTHON-64. Okay, the correct fix for apr_sockaddr_port_get() which works under both Apache 2.0 and 2.2 is to replace the single call in connobject.c, apr_sockaddr_port_get(&port, addr); with port = addr->port; I've added this information to MODPYTHON-78 notes (httpd 2.2 port). Of course that fix only *needs* to be made when 2.2 is completely addressed, but it can be safely rolled in before then too. -- Deron Meranda
Re: 3.2.6 test period - how long do we wait?
On 1/31/06, Jim Gallacher <[EMAIL PROTECTED]> wrote: > apache 2.2. I scanned mod_python for deprecated or removed apr calls and > can find only one (apr_sockaddr_port_get), plus the missing > APR_STATUS_IS_SUCCESS macro. The apr_sockaddr_port_get() call was introduced by me to support IPv6 in modpython with MODPYTHON-64. Looking at the APR 1.0 CHANGES file (accompaning httpd 2.2): *) The following deprecated interfaces have been removed: apr_sockaddr_port_get-> (access directly) I'm trying to review the APR source to see exactly what that means and the conditions under which it's safe. We don't want to accidentally pull the wrong field and have endianness problems. BTW, I want to get this one fixed before release rather than reverting to the pre-MODPYTHON-64 code...or we'll break under IPv6 again. -- Deron Meranda
Re: 3.2.6 test period - how long do we wait?
On Tue, 2006-01-31 at 17:59 +0100, Nicolas Lehuen wrote: > Once again, it seems that no regression have been introduced in 3.2.6 > vs 3.1.4, so we should release it ASAP and try to keep a steady > release rythm afterwards. When we'll get momentum we'll solve a bunch > of problem pretty fast, but it's been a year now that we are paralysed > by perfectionism. What could be worse than leaving our users out there > with the current 3.1.4 version ? +1 :) Rob -- Dr Robert Sanderson Dept of Computer Science, University of Liverpool Home: http://www.csc.liv.ac.uk/~azaroth/ Cheshire: http://www.cheshire3.org/
Re: 3.2.6 test period - how long do we wait?
OK, so shall we schedule the 3.2.x release for 2007, then ? As for the Apache 2.2 version, what if we roll in your suggested patch, Jim, then discover a bunch of problem related to it during the beta tests ? Will we wait until they are all fixed to release the 3.2 version ? Apache 2.2 is quite new so we'll likely to have to squish bugs, due for example to new interaction between Apache filters and mod_python. That's a wild guess but filters have been modified in Apache 2.2 so I'm sure something evil lurks there. Or we could simply forget about making the release one day and tell every user to use the latest snapshot from subversion. Sorry to be like that, but we have users out there that would be perfectly happy with the current state of the 3.2.6 version, and a lot of our answers on the mailing list are "yup, we know this bug, it's already been fixed one year ago, but don't worry, you'll get the bugfix soon enough". Once again, it seems that no regression have been introduced in 3.2.6 vs 3.1.4, so we should release it ASAP and try to keep a steady release rythm afterwards. When we'll get momentum we'll solve a bunch of problem pretty fast, but it's been a year now that we are paralysed by perfectionism. What could be worse than leaving our users out there with the current 3.1.4 version ? Regards, Nicolas 2006/1/31, Jim Gallacher <[EMAIL PROTECTED]>: > I assume we will be doing a 3.2.7 release if Graham's fix for the > ConnectionHandler / MODPYTHON-102 problem works? > > If that is the case I wonder if we should roll in the changes to support > apache 2.2. I scanned mod_python for deprecated or removed apr calls and > can find only one (apr_sockaddr_port_get), plus the missing > APR_STATUS_IS_SUCCESS macro. > > The original macro is: > > #define APR_STATUS_IS_SUCCESS(s) ((s) == APR_SUCCESS \ > || (s) == APR_OS_START_SYSERR + NO_ERROR) > > > The discussion on httpd-dev suggested that this macro should be > substituted with a simple test such as "if (rc != APR_SUCCESS)", and the > '||' condition was not likely used. So that we are making the fewest > possible changes to our current 3.2 codebase, I'd suggest reimplenting > APR_STATUS_IS_SUCCESS in our code, and then removing it 3.3. This will > give us lot's of time as we work on 3.3 to discover if there are any > problems droping the APR_OS_START_SYSERR part of the test. > > Jim >
Re: 3.2.6 test period - how long do we wait?
I assume we will be doing a 3.2.7 release if Graham's fix for the ConnectionHandler / MODPYTHON-102 problem works? If that is the case I wonder if we should roll in the changes to support apache 2.2. I scanned mod_python for deprecated or removed apr calls and can find only one (apr_sockaddr_port_get), plus the missing APR_STATUS_IS_SUCCESS macro. The original macro is: #define APR_STATUS_IS_SUCCESS(s) ((s) == APR_SUCCESS \ || (s) == APR_OS_START_SYSERR + NO_ERROR) The discussion on httpd-dev suggested that this macro should be substituted with a simple test such as "if (rc != APR_SUCCESS)", and the '||' condition was not likely used. So that we are making the fewest possible changes to our current 3.2 codebase, I'd suggest reimplenting APR_STATUS_IS_SUCCESS in our code, and then removing it 3.3. This will give us lot's of time as we work on 3.3 to discover if there are any problems droping the APR_OS_START_SYSERR part of the test. Jim
Re: Segfaults in ConnectionHander FreeBSD (was Re: 3.2.6 test period - how long do we wait?)
David Fraser wrote: Jim Gallacher wrote: Barry Pederson wrote: I think this is the general kind of thing we're looking for though, with some mistaken pointer/memory operation. Too bad we can't write *everything* in python. :( You haven't been following PyPy then? :-) David Well, sure, but I think porting will have to wait until at least mod_python 3.4. :) Jim
Re: 3.2.6 test period - how long do we wait?
Gregory (Grisha) Trubetskoy wrote: On Sun, 29 Jan 2006, Graham Dumpleton wrote: buffer += bufsize; On a second thought - yes, you're right :-) And if he's not then there is a bug in filter_read since that is what it does and it is very similar to _conn_read. Jim
Re: Segfaults in ConnectionHander FreeBSD (was Re: 3.2.6 test period - how long do we wait?)
Jim Gallacher wrote: Barry Pederson wrote: I think this is the general kind of thing we're looking for though, with some mistaken pointer/memory operation. Too bad we can't write *everything* in python. :( You haven't been following PyPy then? :-) David
Re: 3.2.6 test period - how long do we wait?
On Sun, 29 Jan 2006, Graham Dumpleton wrote: buffer += bufsize; On a second thought - yes, you're right :-) Grisha
Re: 3.2.6 test period - how long do we wait?
Grisha wrote .. > > On Sun, 29 Jan 2006, Graham Dumpleton wrote: > > > Grisha wrote .. > >> > >> buffer = bufsize; > >> > > > > I suspect you mean't: > > > > buffer += bufsize; > > buffer = bufsize should be correct because you move the pointer to the > end > of where the bufer was. buffer += bufsize would set it further than you > need it. "buffer" is a char* pointer, specifically the character string buffer inside of the Python string object. "bufsize" is an integer (long) representing the number of bytes read so far. "buffer" is set on the line before that to the start of the internal buffer of the newly resized Python string object. To get to where the next lot of data to be read into, that buffer pointer has to be offset by the number of bytes. Assiging long of bufsize to char* isn't going to work. Thus, still think it is: buffer += bufsize; > > Anyway, this change doesn't help with Mac OS X as it doesn't even get > invoked. > > > > Unlike suggestions by someone else that "self" seemed to be getting > > corrupted, > > it looks fine to me, and code simply crashed down in: > > > > apr_bucket_read(b, &data, &size, APR_BLOCK_READ) > > > > on very first call to it. Thus need to start tracking into Apache itself > and see what > > there may be about bucket structures that isn't correct. This is where > I got to > > last time before I gave up, feeling it wasn't worth the effort at the > time. I'll try > > and build a version of Apache with debug so I can get a better stack > trace. > > The first thing I'd check is for validity of b. Buckets use reference > counting much like Python, so sometimes it's possible for a bucket to > "self-distruct". Starting to delve into the bucket now. Haven't looked at reference count stuff yet, but the b->type object seems to be bogus. This is where the read() function pointer is kept and since it is a bad value it is why it dies. Graham
Re: 3.2.6 test period - how long do we wait?
On Sun, 29 Jan 2006, Graham Dumpleton wrote: Grisha wrote .. buffer = bufsize; I suspect you mean't: buffer += bufsize; buffer = bufsize should be correct because you move the pointer to the end of where the bufer was. buffer += bufsize would set it further than you need it. Anyway, this change doesn't help with Mac OS X as it doesn't even get invoked. Unlike suggestions by someone else that "self" seemed to be getting corrupted, it looks fine to me, and code simply crashed down in: apr_bucket_read(b, &data, &size, APR_BLOCK_READ) on very first call to it. Thus need to start tracking into Apache itself and see what there may be about bucket structures that isn't correct. This is where I got to last time before I gave up, feeling it wasn't worth the effort at the time. I'll try and build a version of Apache with debug so I can get a better stack trace. The first thing I'd check is for validity of b. Buckets use reference counting much like Python, so sometimes it's possible for a bucket to "self-distruct". Grisha
Re: 3.2.6 test period - how long do we wait?
Grisha wrote ..
>
> On Sun, 29 Jan 2006, Jim Gallacher wrote:
>
> > I don't know if this is the answer to the problem, but it looks like
> a bug
> > anyway. In connobject.c starting at line 133:
> >
> >/* time to grow destination string? */
> >if (len == 0 && bytes_read == bufsize) {
> >
> >_PyString_Resize(&result, bufsize + HUGE_STRING_LEN);
> >buffer = PyString_AS_STRING((PyStringObject *) result);
> >buffer += HUGE_STRING_LEN;
> >bufsize += HUGE_STRING_LEN;
> >}
> >
> >
> > It looks like we've just set the buffer pointer to an address somewhere
> > inside the buffer. That can't be good. The buffer pointer should be set
> to
> > the bytes_read position.
>
> ...or bufsize. Of course they are the same, but I think this would read
> cleaner:
>
> if (len == 0 && bytes_read == bufsize) {
>
> _PyString_Resize(&result, bufsize + HUGE_STRING_LEN);
> buffer = PyString_AS_STRING((PyStringObject *) result);
> buffer = bufsize;
> bufsize += HUGE_STRING_LEN;
> }
>
> This bug would garble the data if it's at least twice HUGE_STRING_LEN,
> since the first time around the code would work OK because bufsize would
> equal HUGE_STRING_LEN.
I suspect you mean't:
if (len == 0 && bytes_read == bufsize) {
_PyString_Resize(&result, bufsize + HUGE_STRING_LEN);
buffer = PyString_AS_STRING((PyStringObject *) result);
buffer += bufsize;
bufsize += HUGE_STRING_LEN;
}
Anyway, this change doesn't help with Mac OS X as it doesn't even get invoked.
Unlike suggestions by someone else that "self" seemed to be getting corrupted,
it looks fine to me, and code simply crashed down in:
apr_bucket_read(b, &data, &size, APR_BLOCK_READ)
on very first call to it. Thus need to start tracking into Apache itself and
see what
there may be about bucket structures that isn't correct. This is where I got to
last time before I gave up, feeling it wasn't worth the effort at the time.
I'll try
and build a version of Apache with debug so I can get a better stack trace.
Graham
Re: Segfaults in ConnectionHander FreeBSD (was Re: 3.2.6 test period - how long do we wait?)
Jim Gallacher wrote: Dang, it's frustrating not being able to reproduce this bug in Linux. I suppose it's maybe something to do with different malloc implementations or such. I haven't seen any +1s for OpenBSD, which would be interesting to see since they added some stuff in 3.8 to help catch problems with this sort of thing http://kerneltrap.org/node/5584 Anyone been able to use valgrind or similar with mod_python? I Googled and found a couple old messages from '02 and '04 mentioning attempts to use this, but doesn't sound like much came out of it. I think there's a valgrind port on FreeBSD, so I may give that a try. Barry
Re: 3.2.6 test period - how long do we wait?
On Sun, 29 Jan 2006, Jim Gallacher wrote:
I don't know if this is the answer to the problem, but it looks like a bug
anyway. In connobject.c starting at line 133:
/* time to grow destination string? */
if (len == 0 && bytes_read == bufsize) {
_PyString_Resize(&result, bufsize + HUGE_STRING_LEN);
buffer = PyString_AS_STRING((PyStringObject *) result);
buffer += HUGE_STRING_LEN;
bufsize += HUGE_STRING_LEN;
}
It looks like we've just set the buffer pointer to an address somewhere
inside the buffer. That can't be good. The buffer pointer should be set to
the bytes_read position.
...or bufsize. Of course they are the same, but I think this would read
cleaner:
if (len == 0 && bytes_read == bufsize) {
_PyString_Resize(&result, bufsize + HUGE_STRING_LEN);
buffer = PyString_AS_STRING((PyStringObject *) result);
buffer = bufsize;
bufsize += HUGE_STRING_LEN;
}
This bug would garble the data if it's at least twice HUGE_STRING_LEN,
since the first time around the code would work OK because bufsize would
equal HUGE_STRING_LEN.
Grisha
Re: Segfaults in ConnectionHander FreeBSD (was Re: 3.2.6 test period - how long do we wait?)
Barry Pederson wrote:
I don't know if this is the answer to the problem, but it looks like a
bug anyway. In connobject.c starting at line 133:
/* time to grow destination string? */
if (len == 0 && bytes_read == bufsize) {
_PyString_Resize(&result, bufsize + HUGE_STRING_LEN);
buffer = PyString_AS_STRING((PyStringObject *) result);
buffer += HUGE_STRING_LEN;
bufsize += HUGE_STRING_LEN;
}
It looks like we've just set the buffer pointer to an address
somewhere inside the buffer. That can't be good. The buffer pointer
should be set to the bytes_read position. Perhaps one of you FreeBSD
heads could try the attached patch.
Jim
Index: src/connobject.c
===
--- src/connobject.c(revision 369511)
+++ src/connobject.c(working copy)
@@ -135,7 +135,7 @@
_PyString_Resize(&result, bufsize + HUGE_STRING_LEN);
buffer = PyString_AS_STRING((PyStringObject *) result);
-buffer += HUGE_STRING_LEN;
+buffer += bytes_read;
bufsize += HUGE_STRING_LEN;
}
Sorry, that doesn't seem to fix it. I did a fresh extraction of
mod_python-3.2.6.tgz, applied the patch, did ./configure, make, su, make
install, exit su, cd test, ran test.py - got the same result as before,
with the same core dump apparently.
I really didn't think it would help since a buffer of HUGE_STRING_LEN
(8192) should have been created in the first place. The unit test
wouldn't be reading that many bytes, so I doubt the buffer is getting
resized. All the same it still looks like a bug.
I think this is the general kind of thing we're looking for though, with
some mistaken pointer/memory operation.
Too bad we can't write *everything* in python. :(
---
As I mentioned in another message, I did some experimenting with
disabling other unittests and found if you disable just
"test_fileupload", all the remaining tests including
"test_connectionhandler" pass.
I hadn't forgotten. I'm just trying to understand what might be going on
in the code and I spotted the bug.
If you disable everything except "test_fileupload" and
"test_connectionhandler", then "test_connectionhandler" still crashes.
Dang, it's frustrating not being able to reproduce this bug in Linux.
So I suspect that it's code involved with running "test_fileupload"
(Testing 1 MB file upload support) that's really the source of the
problem, and it's screwing up some part of memory thats only tripped
over later later during the connectionhandler test.
One of the things I'm trying to understand is what has changed since 3.1
that is causing this bug. The only thing different in connobject.c is a
fix to actually return local_ip and local_host (was returning remote_ip
and remote_host), plus makeipaddr and makesockaddr now call the
equivalent apr functions rather than the prior roll-our-own approach.
Neither of these changes should impact the _conn_read.
Jim
Re: 3.2.6 test period - how long do we wait?
Barry Pederson wrote .. > As I mentioned in another message, I did some experimenting with > disabling other unittests and found if you disable just > "test_fileupload", all the remaining tests including > "test_connectionhandler" pass. > > If you disable everything except "test_fileupload" and > "test_connectionhandler", then "test_connectionhandler" still crashes. > > So I suspect that it's code involved with running "test_fileupload" > (Testing 1 MB file upload support) that's really the source of the > problem, and it's screwing up some part of memory thats only tripped > over later later during the connectionhandler test. FWIW, when I came across this on Mac OS X, it wasn't when running the unit tests. In fact the unit tests work fine. It was happening for the very first request after an Apache stop/start. Thus it couldn't have been the result of a specific prior history of requests. Anyway, I am going to give Jim's suggested change a go in a little while and have a bit of a look at the code again to see if I can spot anything obvious. Since though it was on the very first read and no buffer resize was involved, suspecting that it will not help. Graham
Re: 3.2.6 test period - how long do we wait?
I don't know if this is the answer to the problem, but it looks like a
bug anyway. In connobject.c starting at line 133:
/* time to grow destination string? */
if (len == 0 && bytes_read == bufsize) {
_PyString_Resize(&result, bufsize + HUGE_STRING_LEN);
buffer = PyString_AS_STRING((PyStringObject *) result);
buffer += HUGE_STRING_LEN;
bufsize += HUGE_STRING_LEN;
}
It looks like we've just set the buffer pointer to an address somewhere
inside the buffer. That can't be good. The buffer pointer should be set
to the bytes_read position. Perhaps one of you FreeBSD heads could try
the attached patch.
Jim
Index: src/connobject.c
===
--- src/connobject.c(revision 369511)
+++ src/connobject.c(working copy)
@@ -135,7 +135,7 @@
_PyString_Resize(&result, bufsize + HUGE_STRING_LEN);
buffer = PyString_AS_STRING((PyStringObject *) result);
-buffer += HUGE_STRING_LEN;
+buffer += bytes_read;
bufsize += HUGE_STRING_LEN;
}
Sorry, that doesn't seem to fix it. I did a fresh extraction of
mod_python-3.2.6.tgz, applied the patch, did ./configure, make, su, make
install, exit su, cd test, ran test.py - got the same result as before,
with the same core dump apparently.
I think this is the general kind of thing we're looking for though, with
some mistaken pointer/memory operation.
---
As I mentioned in another message, I did some experimenting with
disabling other unittests and found if you disable just
"test_fileupload", all the remaining tests including
"test_connectionhandler" pass.
If you disable everything except "test_fileupload" and
"test_connectionhandler", then "test_connectionhandler" still crashes.
So I suspect that it's code involved with running "test_fileupload"
(Testing 1 MB file upload support) that's really the source of the
problem, and it's screwing up some part of memory thats only tripped
over later later during the connectionhandler test.
Barry
Re: 3.2.6 test period - how long do we wait?
Graham Dumpleton wrote:
On 29/01/2006, at 1:29 AM, Jim Gallacher wrote:
Volodya wrote:
On Fri, Jan 27, 2006 at 10:28:24AM -0500, Gregory (Grisha)
Trubetskoy wrote:
OK, and I see Ron sent a Solaris 10 +1, which is good. I think we
need a FreeBSD +1 - perhaps not necessarily 6.0, but something...
FreeBSD 4.9
In my case PythonConnectionHandler test fails.
Tested on:
Apache/2.0.55 (prefork) mod_python/3.2.6 Python/2.4.2
Apache/2.0.50 (prefork) mod_python/3.2.6 Python/2.3.4
I think this is the same as Barry Pederson's failure.
Which are both similar to:
http://issues.apache.org/jira/browse/MODPYTHON-102
Although in the Mac OS X case, only happened when on secondary port and
not
main listener port.
I have never seen anyone use connection handlers so speculated that
there was
probably no urgency in doing anything about it.
Graham
I don't know if this is the answer to the problem, but it looks like a
bug anyway. In connobject.c starting at line 133:
/* time to grow destination string? */
if (len == 0 && bytes_read == bufsize) {
_PyString_Resize(&result, bufsize + HUGE_STRING_LEN);
buffer = PyString_AS_STRING((PyStringObject *) result);
buffer += HUGE_STRING_LEN;
bufsize += HUGE_STRING_LEN;
}
It looks like we've just set the buffer pointer to an address somewhere
inside the buffer. That can't be good. The buffer pointer should be set
to the bytes_read position. Perhaps one of you FreeBSD heads could try
the attached patch.
Jim
Index: src/connobject.c
===
--- src/connobject.c(revision 369511)
+++ src/connobject.c(working copy)
@@ -135,7 +135,7 @@
_PyString_Resize(&result, bufsize + HUGE_STRING_LEN);
buffer = PyString_AS_STRING((PyStringObject *) result);
-buffer += HUGE_STRING_LEN;
+buffer += bytes_read;
bufsize += HUGE_STRING_LEN;
}
Re: 3.2.6 test period - how long do we wait?
Volodya wrote: On Fri, Jan 27, 2006 at 10:28:24AM -0500, Gregory (Grisha) Trubetskoy wrote: OK, and I see Ron sent a Solaris 10 +1, which is good. I think we need a FreeBSD +1 - perhaps not necessarily 6.0, but something... FreeBSD 4.9 In my case PythonConnectionHandler test fails. Tested on: Apache/2.0.55 (prefork) mod_python/3.2.6 Python/2.4.2 Apache/2.0.50 (prefork) mod_python/3.2.6 Python/2.3.4 I think this is the same as Barry Pederson's failure. Jim
Re: 3.2.6 test period - how long do we wait?
On Fri, Jan 27, 2006 at 10:28:24AM -0500, Gregory (Grisha) Trubetskoy wrote: > > > OK, and I see Ron sent a Solaris 10 +1, which is good. I think we need a > FreeBSD +1 - perhaps not necessarily 6.0, but something... > FreeBSD 4.9 In my case PythonConnectionHandler test fails. Tested on: Apache/2.0.55 (prefork) mod_python/3.2.6 Python/2.4.2 Apache/2.0.50 (prefork) mod_python/3.2.6 Python/2.3.4 I've tried to reproduce this test manually: $ /usr/local/sbin/httpd -k start -f /home/volodya/mod_python-3.2.6/test/conf/test.conf $ GET http://localhost:2758/tests.py 500 Server closed connection without sending any data back $ telnet localhost 2758 telnet localhost 2758 Trying 127.0.0.1... Connected to localhost.mydomain.tld. Escape character is '^]'. Connection closed by foreign host. Child sigfaults immediately after connection being established. logs/error_log: [notice] mod_python: Creating 8 session mutexes based on 256 max processes and 0 max threads. [notice] Apache/2.0.50 (FreeBSD) mod_python/3.2.6 Python/2.3.4 configured -- resuming normal operations [info] Server built: Jul 15 2004 11:25:58 [debug] prefork.c(955): AcceptMutex: flock (default: flock) [notice] child pid 98975 exit signal Segmentation fault (11), possible coredump in /home/volodya/mod_python-3.2.6/test [notice] child pid 98977 exit signal Segmentation fault (11), possible coredump in /home/volodya/mod_python-3.2.6/test Yes, i know that FreeBSD 4.9 is rather old, but this is the only version i have at my disposal.
Re: 3.2.6 test period - how long do we wait?
Maybe I forgot to send this one in: +1 Solaris 10 Sparc Apache-2.0.55 mpm-prefork Python-2.4.2 cheers, Ron Ron Reisor <[EMAIL PROTECTED]> (RWR3) University of Delaware Information Technologies/Network and Systems Services Computing Center/192 South Chapel Street/Newark DE, 19716 pgp finger print: 0D 73 06 6F D3 6A 99 D3 F5 D5 6E FF 3B B9 7C 2C
Re: 3.2.6 test period - how long do we wait?
Gregory (Grisha) Trubetskoy wrote: On Thu, 26 Jan 2006, Jim Gallacher wrote: It seems like any 3.2.6 testing that is going to be done, has been done. I've been kinda swamped with unrelated things past two weeks, so I wasn't paying much attention. Perhaps an e-mail summarizing the +1's so far and a quick vote of the core group on whether this is enough? Grisha Here is the summary of the 3.2.6 testing to date. Let me know if I've missed anyone's tests or you still want to toss you test on the pile. For the 3.2.6b tarball (same as 3.2.6 but with the messed up version string). +1 Gentoo [current], Apache-2.0.55 (mpm-prefork), Python-2.4, gcc-3.4.4 +1 Linux (amd64) Ubuntu Breezy 5.10, apache 2.0.54 mpm-worker python 2.4 +1 Linux Debian sid, apache 2.0.55 (mpm-prefork), python 2.3 +1 Linux Debian sarge, apache 2.0.54 (mpm-worker), python 2.3 +1 Mac OS X 10.3, Apache 2.0.55, python 2.3 +1 Slackware 10.1, Apache 2.0.55 (mpm-prefork), Python 2.4 +1 Windows XP, python 2.4 +1 Windows 2000, python 2.3 For 3.2.6 +1 Fedora Core 4 Apache 2.0.54 Python 2.4.1 +1 Linux Debian sid, apache 2.0.55 (mpm-prefork), python 2.3 +1 Linux Debian sarge, apache 2.0.54 (mpm-worker), python 2.3 +1 MacOS-10.4.4 Apache-2.0.55 mpm-prefork Python-2.4.2 +1 Windows 2000 Server SP4, Python 2.3 +1 Windows XP Pro SP2, Python 2.4 +1 Windows XP Pro SP2 Apache 2.0.54 Python 2.4.2 0 HP Tru64, mpm-worker On the bsd front, Barry Pederson reported a problem with FreeBSD 6.0 but didn't issue an actual -1. FreeBSD and I are still not getting along, but with a small manual tweak of the Makefile I was able to run all the tests successfully. I don't know if there is a configure problem or if it's just my BSD incompetence but I'm willing to issue a shaky +0 for FreeBSD 6.0. Jim
Re: 3.2.6 test period - how long do we wait?
Gregory (Grisha) Trubetskoy wrote: On Thu, 26 Jan 2006, Jim Gallacher wrote: It seems like any 3.2.6 testing that is going to be done, has been done. I've been kinda swamped with unrelated things past two weeks, so I wasn't paying much attention. Perhaps an e-mail summarizing the +1's so far and a quick vote of the core group on whether this is enough? I will gather the info later tonight. Jim
Re: 3.2.6 test period - how long do we wait?
On Thu, 26 Jan 2006, Jim Gallacher wrote: It seems like any 3.2.6 testing that is going to be done, has been done. I've been kinda swamped with unrelated things past two weeks, so I wasn't paying much attention. Perhaps an e-mail summarizing the +1's so far and a quick vote of the core group on whether this is enough? Grisha
Re: 3.2.6 test period - how long do we wait?
Jim Gallacher wrote .. > It seems like any 3.2.6 testing that is going to be done, has been done. > How long do we wait before making a decision for an official release. > > If we don't get cracking on 3.3 soon Graham's gonna fill another couple > of pages on JIRA and we'll never catch up. :) You aren't far wrong. I was actually thinking of sending an email to the list warning about an impending avalanche of JIRA bug reports. This is going to be caused by me going through my module importer issues list and logging separate bug reports for each problem where none already exists in JIRA. Also have potentially others as well I know of but aren't in my module importer issues list at the moment but which I will be adding. Also going to create new bugs or ask for some old bugs to be reopened because the issue was only addressed in version 3.2.6 for mod_python.publisher and not mod_python as a whole. Thus, consider yourself warned. Note that none of what I will be creating issues for should stop 3.2.6 (final) being released. Already accepted that they would be 3.3. Graham
