Re: [Zope-dev] Re: Zope 2.4 crashes -- possible fix identified, other solutions also suggested
>>> "Matthew T. Kromer" wrote > Well, if you want to grab what is probably going to turn in to Python > 2.1.2 from CVS, you can get the release21-maint branch from > :pserver:[EMAIL PROTECTED]:/cvsroot/python package > python. I'd _love_ it if people wanted to do this, once the release candidate is out - I don't have access to a massive number of platforms, so I'd like to know if I've accidentally broken anything. :) Watch this space... -- Anthony Baxter <[EMAIL PROTECTED]> It's never too late to have a happy childhood. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Zope 2.4 crashes -- possible fix identified, other solutions also suggested
>>> "Matthew T. Kromer" wrote > Anthony Baxter is anticipating a Python 2.1.2 beta real soon now (probably > this weekend) so I am going to try to get that into Zope 2.5's binary > releases, although we may put out a Zope 2.5 beta 3 first. This will > include the necessary Python patches to ceval.c to fix the frame bug; after > it hits the streets Python 2.1.2 will become our recommended Python for > Zope. That's still the plan - ripping along through the todo list at the moment. I'm just double-quadruple-triple checking the safety of the patch to make solaris threading not suck as a backport. Note that it's not going to be a beta, but a release candidate - it shouldn't be necessary to have a beta for a bugfix release. Anthony -- Anthony Baxter <[EMAIL PROTECTED]> It's never too late to have a happy childhood. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Zope 2.4 crashes -- possible fix identified, other solutions also suggested
Leonardo Rochael Almeida wrote: > On Tue, 2001-12-18 at 13:44, Matthew T. Kromer wrote: > > Soo... if shutting off GC extends time between crashes for some folks > > from every 15 minutes to 3 times a day, my advise is to shut off GC. > > Now I can really confirm that gc.disable() is enough to avoid the > crashes (no need to recompile python --without-gc). I changed the z2.py to include this, but just as a data point to Matt, ParsedXML is still crashing. ;) Regards, Martijn ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Zope 2.4 crashes -- possible fix identified, other solutions also suggested
Well, if you want to grab what is probably going to turn in to Python 2.1.2 from CVS, you can get the release21-maint branch from :pserver:[EMAIL PROTECTED]:/cvsroot/python package python. This already has the important change to ceval.c in it; but I'm not sure that the rest of the changes for the 2.1.2 release have been finalized. I think Anthony Baxter is going to try to put a beta / release candidate of 2.1.2 up real soon now, but I can't speak for him :) On Wednesday, December 19, 2001, at 10:39 PM, Joseph Wayne Norton wrote: > > Matt - > > If possible, I would prefer to use a source Python 2.1.2 release with > a source zope 2.4.4 ? bugfix release (or create my own from the 2.4 > cvs branch) once the fixes are complete. We do not want to put a 2.5 > release in production at this time. > > Thanks for the update. > > regards, > > - j > > At Wed, 19 Dec 2001 09:25:08 -0500, > Matthew T. Kromer wrote: >> >> The ExtensionClass.h patch has NOT been merged into the branches and >> trunk >> yet pending further review. >> > > just my 2 cents, but we have been using this in production for about 1 > week or more without any troubles. > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Zope 2.4 crashes -- possible fix identified, other solutions also suggested
Matt - If possible, I would prefer to use a source Python 2.1.2 release with a source zope 2.4.4 ? bugfix release (or create my own from the 2.4 cvs branch) once the fixes are complete. We do not want to put a 2.5 release in production at this time. Thanks for the update. regards, - j At Wed, 19 Dec 2001 09:25:08 -0500, Matthew T. Kromer wrote: > > The ExtensionClass.h patch has NOT been merged into the branches and trunk > yet pending further review. > just my 2 cents, but we have been using this in production for about 1 week or more without any troubles. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Zope 2.4 crashes -- possible fix identified, other solutions also suggested
The new compiler fixes that we have to date are actually in 2.4 branch and 2.5 branch as well as the trunk; they're in the lib/python/RestrictedPython/compiler directory; so you could go to http://cvs.zope.org and just pull down that directory (optionally pulling down your specific branch too). Pythonlabs is working on a few more cases which I expect will be done and checked in RSN; then we'll backport those to the branches so you don't have to install a new python to get the fixes with Zope. Anthony Baxter is anticipating a Python 2.1.2 beta real soon now (probably this weekend) so I am going to try to get that into Zope 2.5's binary releases, although we may put out a Zope 2.5 beta 3 first. This will include the necessary Python patches to ceval.c to fix the frame bug; after it hits the streets Python 2.1.2 will become our recommended Python for Zope. The ExtensionClass.h patch has NOT been merged into the branches and trunk yet pending further review. - Original Message - From: "Joseph Wayne Norton" <[EMAIL PROTECTED]> To: "Matthew T. Kromer" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Tuesday, December 18, 2001 7:59 PM Subject: Re: [Zope-dev] Re: Zope 2.4 crashes -- possible fix identified, other solutions also suggested > > Matt et al - > > We are using python 2.1.1 with pymalloc disabled and gc enabled with > zope 2.4.3 on the solaris platform. We are currently seeing only 1 or > 2 restarts a day for the zope/zeo clients. Fortunately, we have not > experienced any trouble with the zeo server. > > We have applied the a) extensionclass bugfix and I'm also planning to > apply the b) frame setup bugfix later this week. We feel that patch > a) has reduced the number of restarts ... but the site activity has > descreased as well so it is difficult to really confirm. > > 1) I would like to know if any patches are available yet for the c) > compiler stack size bug. When and if this patch is available, would > it require a new python installation, a new zope installation, or > both? > > 2) Are there any other related patches/comments with respect to python > 2.1.1 and zope 2.4.3 and this issue? > > thanks, > > - joe n. > > > At Tue, 18 Dec 2001 10:44:33 -0500, > Matthew T. Kromer wrote: > > > > Well, one suspicion I have is that (aside from memory corruption caused > > by the compiler sack size bugs and the frame setup bug in 2.1 when > > handling exceptions) ExtensionClasses are providing bogus data to > > modules which aren't checking the flags to see if the GC fields are > > populated. > > > > Some of the people who have tried the modified extensionclass.h which > > pads out the type object to align it with the 2.1 type object THINK they > > have seen a reduction in crashes, but these same folks also have not > > applied fixes for the two known bugs. > > > > Soo... if shutting off GC extends time between crashes for some folks > > from every 15 minutes to 3 times a day, my advise is to shut off GC. > > > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Zope 2.4 crashes -- possible fix identified, other solutions also suggested
Matt et al - We are using python 2.1.1 with pymalloc disabled and gc enabled with zope 2.4.3 on the solaris platform. We are currently seeing only 1 or 2 restarts a day for the zope/zeo clients. Fortunately, we have not experienced any trouble with the zeo server. We have applied the a) extensionclass bugfix and I'm also planning to apply the b) frame setup bugfix later this week. We feel that patch a) has reduced the number of restarts ... but the site activity has descreased as well so it is difficult to really confirm. 1) I would like to know if any patches are available yet for the c) compiler stack size bug. When and if this patch is available, would it require a new python installation, a new zope installation, or both? 2) Are there any other related patches/comments with respect to python 2.1.1 and zope 2.4.3 and this issue? thanks, - joe n. At Tue, 18 Dec 2001 10:44:33 -0500, Matthew T. Kromer wrote: > > Well, one suspicion I have is that (aside from memory corruption caused > by the compiler sack size bugs and the frame setup bug in 2.1 when > handling exceptions) ExtensionClasses are providing bogus data to > modules which aren't checking the flags to see if the GC fields are > populated. > > Some of the people who have tried the modified extensionclass.h which > pads out the type object to align it with the 2.1 type object THINK they > have seen a reduction in crashes, but these same folks also have not > applied fixes for the two known bugs. > > Soo... if shutting off GC extends time between crashes for some folks > from every 15 minutes to 3 times a day, my advise is to shut off GC. > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Zope 2.4 crashes -- possible fix identified, other solutions also suggested
Leonardo Rochael Almeida wrote: >On Tue, 2001-12-18 at 13:44, Matthew T. Kromer wrote: > >>Soo... if shutting off GC extends time between crashes for some folks >>from every 15 minutes to 3 times a day, my advise is to shut off GC. >> > >Now I can really confirm that gc.disable() is enough to avoid the >crashes (no need to recompile python --without-gc). > >And as far as we could notice, it didn't start leaking like crazy. We >haven't been able to identify any noticeable increase in memory >consumption. If it is leaking we will need some more time to notice it. > >Cheers, Leo > Keep in mind that the leaks you may experience are directly related to what code you run, and whether or not they introduce cycles. Some of the restricted python compiler code did/does create cycles under the assumption that the GC would break them. But if you dont lean on code that uses RestrictedPython too much, you can live with slow leaks elsewhere. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Zope 2.4 crashes -- possible fix identified,other solutions also suggested
On Tue, 2001-12-18 at 13:44, Matthew T. Kromer wrote: > Soo... if shutting off GC extends time between crashes for some folks > from every 15 minutes to 3 times a day, my advise is to shut off GC. Now I can really confirm that gc.disable() is enough to avoid the crashes (no need to recompile python --without-gc). And as far as we could notice, it didn't start leaking like crazy. We haven't been able to identify any noticeable increase in memory consumption. If it is leaking we will need some more time to notice it. Cheers, Leo -- Ideas don't stay in some minds very long because they don't like solitary confinement. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Zope 2.4 crashes -- possible fix identified, other solutions also suggested
Jeremy Hylton wrote: >>"MTK" == Matthew T Kromer <[EMAIL PROTECTED]> writes: >> > > MTK> A side effect of shutting off the garbage collector is that you > MTK> can have some storage leaks. We're working on being able to > MTK> re-enable the garbage collector so that you don't exhaust > MTK> memory over time. > >Do you have any more idea about what shutting the garbage collector >off achieves? In practice, the garbage collector's most common effect >is to turn latent bugs into manifest bugs; a bug has trashed part of >memory and the garbage collector just happens to find it first. If >you turn GC off in these cases, you run a little longer, but you're >running with corrupted memory. > >Jeremy > Well, one suspicion I have is that (aside from memory corruption caused by the compiler sack size bugs and the frame setup bug in 2.1 when handling exceptions) ExtensionClasses are providing bogus data to modules which aren't checking the flags to see if the GC fields are populated. Some of the people who have tried the modified extensionclass.h which pads out the type object to align it with the 2.1 type object THINK they have seen a reduction in crashes, but these same folks also have not applied fixes for the two known bugs. Soo... if shutting off GC extends time between crashes for some folks from every 15 minutes to 3 times a day, my advise is to shut off GC. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Zope 2.4 crashes -- possible fix identified, other solutions also suggested
>>> Jeremy Hylton wrote > Do you have any more idea about what shutting the garbage collector > off achieves? In practice, the garbage collector's most common effect > is to turn latent bugs into manifest bugs; a bug has trashed part of > memory and the garbage collector just happens to find it first. If > you turn GC off in these cases, you run a little longer, but you're > running with corrupted memory. Sorry I haven't been keeping up with the zope-* lists of late - this is what I've found as well. Something, and I strongly suspect it's inside the Zope C code, is playing jumpy-jumpy-stomp-stomp on bits of memory. The garbage collector is hitting this corrupted data and dying. I've posted before about the structure I've found that's corrupted (it's _always_ the same structure) but I've not yet been able to track down what it is. For us, the "fix" has been to run more zeo clients behind a loadbalancer, so that when one crashes out (about every 10-12 hours for us) things keep working, and the zopecontrol script restarts it. Anthony ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: Zope 2.4 crashes -- possible fix identified, other solutions also suggested
> "MTK" == Matthew T Kromer <[EMAIL PROTECTED]> writes: MTK> A side effect of shutting off the garbage collector is that you MTK> can have some storage leaks. We're working on being able to MTK> re-enable the garbage collector so that you don't exhaust MTK> memory over time. Do you have any more idea about what shutting the garbage collector off achieves? In practice, the garbage collector's most common effect is to turn latent bugs into manifest bugs; a bug has trashed part of memory and the garbage collector just happens to find it first. If you turn GC off in these cases, you run a little longer, but you're running with corrupted memory. Jeremy ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
