Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?
> Oliver Bleutgen writes: > > Shouldn't tags for creating sql-statements in ZSQL > > (like dtml-sqltest) quote the variables by default in order > > to prevent unexpected conversions by the database? > Did you check, that standard SQL supports quoted > names? First, I meant double quotes (") not single one's ('), in case that wasn't clear. I know that for mysql, postgres, oracle there are column names which will surely break dtml-sqltest as it is implemented now. Either lowercase letters (oracle, as you also noted) or uppercase (the other two). I guess sql-keywords and special characters as column names - ok, not a very clever idea - will break sqltest on all of them. At least I didn't find a way to get it to work without patching zope. Searching the web I find various hints that many odbc,jdbc adapters and sql-db frontends use quoted names per default (psql). Unfortunately I don't want to pay to the ANSI-commitee for reading into the sql-standard, but I'm pretty sure every newer database supports that. > Even if it does, the proposed change will probably break > lots of existing code. Right, how about adding a argument like, let's say "sql_quote"? ;) Unfortunatly, I fear I'm have not enough knowledge in zope's internals to offer a clean patch. > I fear, in our projects, the fact that case does not matter > has been widely used. > We would have to change hundreds of SQL methods > after the proposed change. > For new projects, I would like to have it your way. > However, this will be difficult to achieve. I would like to see quoting as a standard, but you have a point. Maybe sql_quote is the way to go. cheers, oliver ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?
Oliver Bleutgen writes: > Shouldn't tags for creating sql-statements in ZSQL > (like dtml-sqltest) quote the variables by default in order > to prevent unexpected conversions by the database? Did you check, that standard SQL supports quoted names? Even if it does, the proposed change will probably break lots of existing code. I fear, in our projects, the fact that case does not matter has been widely used. We would have to change hundreds of SQL methods after the proposed change. For new projects, I would like to have it your way. However, this will be difficult to achieve. Dieter ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?
> Some databases convert them to uppercase! > Oracle is a prominent example! > It took me once some ours to locate a bug resulting > from this "feature". This reminds me of another post I did some time before, but where nobody replied. Shouldn't tags for creating sql-statements in ZSQL (like dtml-sqltest) quote the variables by default in order to prevent unexpected conversions by the database? I had to patch Zope in order to be able do queries on a postgres db with capital column-names, because postgres converts all unquoted columnnames to lowercase. cheers, oliver ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?
Cees de Groot writes: > Jon Prettyman <[EMAIL PROTECTED]> said: > >If you are using ZSQLMethods and your database returns column names in > >one case (UPPER) but you reference them with another (lower) it > >appears that instances of SQLAlias get leaked. > > > Thanks for the info (and another reason to keep SQL names in lowercase ;-)). Be careful! Some databases convert them to uppercase! Oracle is a prominent example! It took me once some ours to locate a bug resulting from this "feature". Dieter ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?
Jon Prettyman <[EMAIL PROTECTED]> said: >If you are using ZSQLMethods and your database returns column names in >one case (UPPER) but you reference them with another (lower) it >appears that instances of SQLAlias get leaked. > Thanks for the info (and another reason to keep SQL names in lowercase ;-)). -- Cees de Groot http://www.cdegroot.com <[EMAIL PROTECTED]> GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD 1986 F303 937F E098 9E8B ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?
If you are using ZSQLMethods and your database returns column names in one case (UPPER) but you reference them with another (lower) it appears that instances of SQLAlias get leaked. Brian Lloyd has put Jim Fulton's fix in the 2.2.5b release that's up on the site. We were seeing thousands of leaked instances of SQLAlias causing our memory usage to skyrocket. -jon [EMAIL PROTECTED] (Cees de Groot) writes: > Jon Prettyman <[EMAIL PROTECTED]> said: > >Are you using ZSQLMethods by any chance? > > > Yes, why? (I think I've fixed /this/ problem, but if ZSQLMethods will > cause similar behaviour I'd like to be prepared :-)) > > -- ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?
Jon Prettyman <[EMAIL PROTECTED]> said: >Are you using ZSQLMethods by any chance? > Yes, why? (I think I've fixed /this/ problem, but if ZSQLMethods will cause similar behaviour I'd like to be prepared :-)) -- Cees de Groot http://www.cdegroot.com <[EMAIL PROTECTED]> GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD 1986 F303 937F E098 9E8B ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?
[EMAIL PROTECTED] said: > I've made zope inaccessible two different ways: > - passing bad data to ZDGChart - sending mail when MailHost is down > (i.e. the host it is supposed to connect to is not available) It seems a bug in my virtual hosting code (a Python method that acts as the AccessRule for the root) pushes an empty string on the TraversalRequestNameStack. Could that be the cause? -- Cees de Groot http://www.cdegroot.com <[EMAIL PROTECTED]> GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD 1986 F303 937F E098 9E8B ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?
Are you using ZSQLMethods by any chance? -jon [EMAIL PROTECTED] (Cees de Groot) writes: > Our production Zope machine has developed a habit over Christmas where > it starts eating CPU and RAM (one Python thread on 100% CPU time, and > I've seen kernel messages indicating that Zope was killed because it > hit an out-of-memory condition which means it sat on ~500Mb RAM). > > I'm completely in the dark about what causes this (probably some of > my own code, I installed some new bits a couple of days ago), but what's > worse: I really don't know how to set about finding the problematic > code. Is there a way to find out what code a thread is running? > > -- > Cees de Groot http://www.cdegroot.com <[EMAIL PROTECTED]> > GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD 1986 F303 937F E098 9E8B > > ___ > Zope maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?
[EMAIL PROTECTED] said: > Turning on detailed logging at startup might help too... read about > the -M argument in the z2.py script and read my Howto regarding "How > To Track Down Hangs" at http://www.zope.org/Members/mcdonc/HowTos/ > DEBUG-LOG Thanks! That looks promising! -- Cees de Groot http://www.cdegroot.com <[EMAIL PROTECTED]> GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD 1986 F303 937F E098 9E8B ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?
Dieter Maurer <[EMAIL PROTECTED]> said: >Cees de Groot writes: > > ... excessive CPU/memory consumption ... >If you use Zope 2.2.x, you can try to access the debug tools >in the "Control Panel". They will show you, which request >is active (to more easily reproduce and analyse the problem) >and how the reference counts are changed. > The problem is, apparently the thing gets in a really tight loop - one threads blocks the rest of Zope, so it is quite hard to use these tools (especially not when the whole thing is not - yet - reproducable, sometimes it happens in an hour, sometimes after 12 hours; a bit hard to keep staring at the screen for 12 hours :-)). As a bit of an act of despair, I'm currently logging Zope's CPU usage every 5 seconds in a log file, hopefully I can match a rise in CPU time with some request in the server logs. -- Cees de Groot http://www.cdegroot.com <[EMAIL PROTECTED]> GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD 1986 F303 937F E098 9E8B ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?
Turning on detailed logging at startup might help too... read about the -M argument in the z2.py script and read my Howto regarding "How To Track Down Hangs" at http://www.zope.org/Members/mcdonc/HowTos/DEBUG-LOG "Cees de Groot" <[EMAIL PROTECTED]> wrote in message 92ghr3$jf3$[EMAIL PROTECTED]">news:92ghr3$jf3$[EMAIL PROTECTED]... > Dieter Maurer <[EMAIL PROTECTED]> said: > >Cees de Groot writes: > > > ... excessive CPU/memory consumption ... > >If you use Zope 2.2.x, you can try to access the debug tools > >in the "Control Panel". They will show you, which request > >is active (to more easily reproduce and analyse the problem) > >and how the reference counts are changed. > > > The problem is, apparently the thing gets in a really tight loop - one > threads blocks the rest of Zope, so it is quite hard to use these > tools (especially not when the whole thing is not - yet - reproducable, > sometimes it happens in an hour, sometimes after 12 hours; a bit hard > to keep staring at the screen for 12 hours :-)). > > As a bit of an act of despair, I'm currently logging Zope's CPU usage > every 5 seconds in a log file, hopefully I can match a rise in CPU time > with some request in the server logs. > > > -- > Cees de Groot http://www.cdegroot.com <[EMAIL PROTECTED]> > GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD 1986 F303 937F E098 9E8B > > ___ > Zope maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope-dev ) > > ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?
Cees de Groot writes: > ... excessive CPU/memory consumption ... If you use Zope 2.2.x, you can try to access the debug tools in the "Control Panel". They will show you, which request is active (to more easily reproduce and analyse the problem) and how the reference counts are changed. Dieter ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
[Zope] Zope eating CPU/RAM - how do I find the culprit?
Our production Zope machine has developed a habit over Christmas where it starts eating CPU and RAM (one Python thread on 100% CPU time, and I've seen kernel messages indicating that Zope was killed because it hit an out-of-memory condition which means it sat on ~500Mb RAM). I'm completely in the dark about what causes this (probably some of my own code, I installed some new bits a couple of days ago), but what's worse: I really don't know how to set about finding the problematic code. Is there a way to find out what code a thread is running? -- Cees de Groot http://www.cdegroot.com <[EMAIL PROTECTED]> GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD 1986 F303 937F E098 9E8B ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )