Re: [Zope-dev] still segfaults, this time no ZMySQLDA

[Casey Duncan]

On Thursday 28 February 2002 02:51 pm, Leonardo Rochael Almeida allegedly 
wrote:
[snip]
> When you see the result page, you know that your pythonScripts are
> recompiled (of course it won't take that long unless you have an 800+MiB
> Data.fs crammed full of python scripts :-)
>
> Cheers, Leo

I hope those weren't all coded in a textarea 8*). Ouch.

/---\
  Casey Duncan, Sr. Web Developer
  National Legal Aid and Defender Association
  [EMAIL PROTECTED]
\---/

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] still segfaults, this time no ZMySQLDA

[Leonardo Rochael Almeida]

On Thu, 2002-02-28 at 16:09, Leonardo Rochael Almeida wrote:
> On Thu, 2002-02-28 at 10:55, Matthew T. Kromer wrote:
> > 
> > On Wednesday, February 27, 2002, at 06:49 PM, Leonardo Rochael Almeida 
> > wrote:
> > 
> > > So, what should I try now?
> > >
> > 
> > Did -t 1 help?  If not, do you feel comfortable hooking up gdb  and 
> > getting a traceback?
> 
> Funny enough, running with -t 1 solved it.
> 
> /me starts looking sideways to ZMySQLDA again, since it's not the new
> beta version...
> 
> The performance, understandably, is not the greatest, but it's tolerable
> (where tolerable means the client hasn't complained yet).
> 
> I'll start looking at ways to recompile my python scripts using another
> ZEO Client. I think I'll catalog all of them (using a very small
> subtransaction threshould) and use a "REQUEST.write()" in a PyScript to
> list them.
> 

COOL!!

I found a kickass way to recompile all PythonScripts without bringing
down the machine and writing the least amount of code ever possible!

Short version:

Create a python script in Zope root that recompiles "context" and
catalog all python scripts using that as a metadata.

Long version:

put the following python script in your root:
=
## Script (Python) "recompile"
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=
##title=
##
# recompile the thing
context.write(context.read())

# return something to waste catalog space :-)
from string import join
return "recompiled %s\n" % join(context.getPhysicalPath(),"/")
=

Now put a ZCatalog in your root, call it anything, say, pyScripts. Go to
the "Indexex" tab and remove all indexes but the "path" index. Go back
and go to the "Metadata" tab; remove all metadata and insert "recompile"
and getPhysicalPath (this one is completely optional). Go to the
Advanced tab and set the subtransaction threshould to a number that
looks reasonable to you (low enough so as not to  crash your machine
:-). Go to the "Find Objects" tab, make sure that only "Script (Python)"
is selected (verify, especially, that "All types" isn't selected) and
click "Find and Catalog".

wait...

wait...

wait some more...

go get a coke or a coffee or something...

wait a little more...

When you see the result page, you know that your pythonScripts are
recompiled (of course it won't take that long unless you have an 800+MiB
Data.fs crammed full of python scripts :-)

Cheers, Leo

-- 
Ideas don't stay in some minds very long because they don't like
solitary confinement.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] still segfaults, this time no ZMySQLDA

[Matthew T. Kromer]

On Thursday, February 28, 2002, at 02:09 PM, Leonardo Rochael Almeida 
wrote:
>
> Funny enough, running with -t 1 solved it.
>
> /me starts looking sideways to ZMySQLDA again, since it's not the new
> beta version...
>

Good!

Based on the fact that I'm not being innundated with "I'm still 
crashing!" complaints, I'm going to *guess* that -t 1 isolated the bad 
module, which was probably being activated even though you didn't think 
it was.  Maybe something traversed to it, or activated it in some other 
way.

I don't think there is a thread-sensitive gremlin in Zope 2.5, but I'm 
asking all parties to keep an eye open for a vanilla 2.5 (or 2.4.4) 
crash that -t 1 solves.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] still segfaults, this time no ZMySQLDA

[Leonardo Rochael Almeida]

On Thu, 2002-02-28 at 10:55, Matthew T. Kromer wrote:
> 
> On Wednesday, February 27, 2002, at 06:49 PM, Leonardo Rochael Almeida 
> wrote:
> 
> > So, what should I try now?
> >
> 
> Did -t 1 help?  If not, do you feel comfortable hooking up gdb  and 
> getting a traceback?

Funny enough, running with -t 1 solved it.

/me starts looking sideways to ZMySQLDA again, since it's not the new
beta version...

The performance, understandably, is not the greatest, but it's tolerable
(where tolerable means the client hasn't complained yet).

I'll start looking at ways to recompile my python scripts using another
ZEO Client. I think I'll catalog all of them (using a very small
subtransaction threshould) and use a "REQUEST.write()" in a PyScript to
list them.

-- 
Ideas don't stay in some minds very long because they don't like
solitary confinement.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] still segfaults, this time no ZMySQLDA

[Matthew T. Kromer]

On Wednesday, February 27, 2002, at 06:49 PM, Leonardo Rochael Almeida 
wrote:

> So, what should I try now?
>

Did -t 1 help?  If not, do you feel comfortable hooking up gdb  and 
getting a traceback?


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] still segfaults, this time no ZMySQLDA

[Anthony Baxter]

>>> Leonardo Rochael Almeida wrote
> So, what should I try now?

I've been unable to keep up with email lately (no decent net
connection) but can I ask if you've got a core dump at all?
If so, where was it? Inside the garbage collector, I'm guessing...
If so, what was the object _before_ the corrupted object in memory?
(Had I thought to look at this when I had similar problems, I 
would have picked up the stupid stacksize bug much earlier...)
If you haven't recompiled the scripts after upgrading to a working
PythonScripts, you'll still see crashes.

When you run with '-D', do you get a lot of "Script (Python) foo needs to
be recompiled"? If the problem is that you can't recompile the scripts,
why not shut it all down, manually mount the ZODB, and recompile 10 
scripts at a time? (This is a not-particularly difficult task - I can
give a sample bit of source if it would help...)

Anthony


-- 
Anthony Baxter <[EMAIL PROTECTED]>   
It's never to late to have a happy childhood.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] still segfaults, this time no ZMySQLDA

[Leonardo Rochael Almeida]

So, what should I try now?

On Wed, 2002-02-27 at 16:18, Matthew T. Kromer wrote:
> Leonardo Rochael Almeida wrote:
> 
> >On Tue, 2002-02-26 at 21:55, Matthew T. Kromer wrote:
> >
> >>in lib/python/Products/PythonScripts/PythonScripts.py, try changing 
> >>Script_magic to be 4, and see if that helps.
> >>
> >
> >Nope, time between restart after the change and 1st crash was aprox.
> >3m15s.
> >
> 
> Rats!  I was hoping that would flush things out, but wasn't expecting it 
> to work.
> 
> >>FYI, some of this is written up in
> >>
> >>  http://www.zope.org/Members/matt/StabilityHOWTO
> >>
> >
> >hmm, I didn't know that page. It mentions trying with the -t 1. In this
> >case, what is the real total number of threads? is it one for Medusa and
> >one for handling Zope requests or is it just one overall? In this case I
> >might be able to run it under Valgrind after all.
> >
> >Should I bother trying a different ExtensionClass.h?
> >
> >Cheers, Leo
> >
> 
> With -t 1, you're still running with more than one real thread, but only 
> 1 worker thread is processing Zope requests.  Most of the time when 
> there is a threading issue, the worker Zope threads are the ones that do 
> the potentially thread-unsafe work, so running with -t 1 will prevent a 
> thread switch from occurring while an exposure could take place.
> 
> You don't need to bother with the different ExtensionClass.h; the magic 
> in my special version is NOT necessary to apply, and should not make 
> your system any more stable.

-- 
Ideas don't stay in some minds very long because they don't like
solitary confinement.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] still segfaults, this time no ZMySQLDA

[Dieter Maurer]

Leonardo Rochael Almeida writes:
 > ...
 > I'm still looking for a way to recompile them all without
 > sending the machine (1GiB physical RAM) into a swap storm which would
 > ultimately crash Zope or the machine and keep it from commiting the
 > transaction that caused all the trouble in the first place (one of these
 > days I'll refactor the subtransaction code out of the ZCatalog and into
 > a ZopeFind that will accept an action parameter, along with a
 > subtransaction one, to perform these kinds of global maintenance in a
 > reasonable way).
You may commit the transaction after several recompilations (as you
suggest). You can see how to do this in e.g. "emulateRedirect" at

  


Dieter

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] still segfaults, this time no ZMySQLDA

[Matthew T. Kromer]

Leonardo Rochael Almeida wrote:

>On Tue, 2002-02-26 at 21:55, Matthew T. Kromer wrote:
>
>>in lib/python/Products/PythonScripts/PythonScripts.py, try changing 
>>Script_magic to be 4, and see if that helps.
>>
>
>Nope, time between restart after the change and 1st crash was aprox.
>3m15s.
>

Rats!  I was hoping that would flush things out, but wasn't expecting it 
to work.

>>FYI, some of this is written up in
>>
>>  http://www.zope.org/Members/matt/StabilityHOWTO
>>
>
>hmm, I didn't know that page. It mentions trying with the -t 1. In this
>case, what is the real total number of threads? is it one for Medusa and
>one for handling Zope requests or is it just one overall? In this case I
>might be able to run it under Valgrind after all.
>
>Should I bother trying a different ExtensionClass.h?
>
>Cheers, Leo
>

With -t 1, you're still running with more than one real thread, but only 
1 worker thread is processing Zope requests.  Most of the time when 
there is a threading issue, the worker Zope threads are the ones that do 
the potentially thread-unsafe work, so running with -t 1 will prevent a 
thread switch from occurring while an exposure could take place.

You don't need to bother with the different ExtensionClass.h; the magic 
in my special version is NOT necessary to apply, and should not make 
your system any more stable.

-- 
Matt Kromer
Zope Corporation  http://www.zope.com/ 




___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] still segfaults, this time no ZMySQLDA

[Leonardo Rochael Almeida]

On Tue, 2002-02-26 at 21:55, Matthew T. Kromer wrote:
> in lib/python/Products/PythonScripts/PythonScripts.py, try changing 
> Script_magic to be 4, and see if that helps.

Nope, time between restart after the change and 1st crash was aprox.
3m15s.

> FYI, some of this is written up in
> 
>   http://www.zope.org/Members/matt/StabilityHOWTO

hmm, I didn't know that page. It mentions trying with the -t 1. In this
case, what is the real total number of threads? is it one for Medusa and
one for handling Zope requests or is it just one overall? In this case I
might be able to run it under Valgrind after all.

Should I bother trying a different ExtensionClass.h?

Cheers, Leo

-- 
Ideas don't stay in some minds very long because they don't like
solitary confinement.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] still segfaults, this time no ZMySQLDA

[Matthew T. Kromer]

in lib/python/Products/PythonScripts/PythonScripts.py, try changing 
Script_magic to be 4, and see if that helps.

FYI, some of this is written up in

  http://www.zope.org/Members/matt/StabilityHOWTO


On Tuesday, February 26, 2002, at 07:15 PM, Leonardo Rochael Almeida 
wrote:

> On Tue, 2002-02-26 at 20:11, Leonardo Rochael Almeida wrote:
>> [...]
>> The options I think I have left:
>> * Try the new ExtensionClass.h
>
> Which one, by the way? the one in HEAD, the one in 2.5 branch or a
> different one altogether? Does it have any impact in the objects in the
> ZODB? Can I run a ZEO Client with it against a ZEO Server with a
> different ExtensionClass.h?
>
>> * Run Zope under a memory analiser such as Valgrind (which might not be
>> feasible if the runtimes are too slow):
>> http://devel-home.kde.org/~sewardj/
>
> Scratch that. Valgrind doesn't do threads. Anyone knows any
> alternatives?
>
> --
> Ideas don't stay in some minds very long because they don't like
> solitary confinement.
>
>
> ___
> Zope-Dev maillist  -  [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope-dev
> **  No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope )


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )

Re: [Zope-dev] still segfaults, this time no ZMySQLDA

[Leonardo Rochael Almeida]

On Tue, 2002-02-26 at 20:11, Leonardo Rochael Almeida wrote:
> [...]
> The options I think I have left:
> * Try the new ExtensionClass.h

Which one, by the way? the one in HEAD, the one in 2.5 branch or a
different one altogether? Does it have any impact in the objects in the
ZODB? Can I run a ZEO Client with it against a ZEO Server with a
different ExtensionClass.h?

> * Run Zope under a memory analiser such as Valgrind (which might not be
> feasible if the runtimes are too slow):
> http://devel-home.kde.org/~sewardj/

Scratch that. Valgrind doesn't do threads. Anyone knows any
alternatives?

-- 
Ideas don't stay in some minds very long because they don't like
solitary confinement.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )

[Zope-dev] still segfaults, this time no ZMySQLDA

[Leonardo Rochael Almeida]

Hi, It's me and my instability problems again :-)

The system:
Zope-2.4.4b1 (binary linux-x86 from zope.org)
Python-2.1.2 (see above)

Products (excluding Zope defaults and ZClass based ones):
LocalFS 
Pagina0_Base (Hiperlógica product)
Pagina1_Base (Hiperlógica product)
Renderable
exUserFolder

It has an 800+MiB Data.fs (when packed) that is crammed full of python
scripts. I'm still looking for a way to recompile them all without
sending the machine (1GiB physical RAM) into a swap storm which would
ultimately crash Zope or the machine and keep it from commiting the
transaction that caused all the trouble in the first place (one of these
days I'll refactor the subtransaction code out of the ZCatalog and into
a ZopeFind that will accept an action parameter, along with a
subtransaction one, to perform these kinds of global maintenance in a
reasonable way).

None of the products mentioned above use C compiled modules. I disabled
all the use of mysql on this instance so that I could remove ZMySQLDA
and faked the authentication system so that it's usage pattern remained
the same even without mysql. After that I reenabled the cycle-gc.

The system started crashing about every 5 minutes.

As much as I'd have liked pointing a finger at ZMySQLDA, start the
process of migration to PostgreSQL and consider this problem solved once
and for all, I have finally come to the conclusion that there is no one
else to blame but Zope and Python themselves. I have since reenabled
ZMySQLDA use by the site and disabled the cycle-gc.

The options I think I have left:
* Try the new ExtensionClass.h
* Run Zope under a memory analiser such as Valgrind (which might not be
feasible if the runtimes are too slow):
http://devel-home.kde.org/~sewardj/
* Try migrating to Zope-2.5.1b1

The situation is starting to look desperate. We now have to restart the
server twice a day to keep the machine from swapping. This can't go on
any longer. Any suggestion is apreciated.

-- 
Ideas don't stay in some minds very long because they don't like
solitary confinement.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )