Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?

2001-01-04 Thread Oliver Bleutgen

   Some databases convert them to uppercase!
   Oracle is a prominent example!

   It took me once some ours to locate a bug resulting
   from this "feature".

This reminds me of another post I did some time before, but
where nobody replied.

Shouldn't tags for creating sql-statements in ZSQL
(like dtml-sqltest) quote the variables by default in order
to prevent unexpected conversions by the database?

I had to patch Zope in order to be able do queries on
a postgres db with capital column-names, because postgres 
converts all unquoted columnnames to lowercase.


cheers,
oliver



___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?

2001-01-04 Thread Dieter Maurer

Oliver Bleutgen writes:
  Shouldn't tags for creating sql-statements in ZSQL
  (like dtml-sqltest) quote the variables by default in order
  to prevent unexpected conversions by the database?
Did you check, that standard SQL supports quoted
names?

Even if it does, the proposed change will probably break
lots of existing code.
I fear, in our projects, the fact that case does not matter
has been widely used.
We would have to change hundreds of SQL methods
after the proposed change.

For new projects, I would like to have it your way.
However, this will be difficult to achieve.


Dieter

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?

2001-01-04 Thread Oliver Bleutgen

 Oliver Bleutgen writes:
   Shouldn't tags for creating sql-statements in ZSQL
   (like dtml-sqltest) quote the variables by default in order
   to prevent unexpected conversions by the database?
 Did you check, that standard SQL supports quoted
 names?

First, I meant double quotes (") not single one's ('),
in case that wasn't clear.

I know that for mysql, postgres, oracle there are column names
which will surely break dtml-sqltest as it is implemented now.
Either lowercase letters (oracle, as you also noted) or uppercase
(the other two). I guess sql-keywords and special characters as
column names - ok, not a very clever idea - will break sqltest
on all of them. At least I didn't find a way to get it to 
work without patching zope.

Searching the web I find various hints that many odbc,jdbc adapters
and sql-db frontends use quoted names per default (psql).
Unfortunately I don't want to pay to the ANSI-commitee for reading
into the sql-standard, but I'm pretty sure every newer database
supports that.


 Even if it does, the proposed change will probably break
 lots of existing code.

Right, how about adding a argument like, let's say "sql_quote"? ;)
Unfortunatly, I fear I'm have not enough knowledge in zope's 
internals to offer a clean patch.

 I fear, in our projects, the fact that case does not matter
 has been widely used.
 We would have to change hundreds of SQL methods
 after the proposed change.
 For new projects, I would like to have it your way.
 However, this will be difficult to achieve.

I would like to see quoting as a standard, but you have a point.
Maybe sql_quote is the way to go.


cheers,
oliver



___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?

2001-01-03 Thread Dieter Maurer

Cees de Groot writes:
  Jon Prettyman [EMAIL PROTECTED] said:
  If you are using ZSQLMethods and your database returns column names in
  one case (UPPER) but you reference them with another (lower) it
  appears that instances of SQLAlias get leaked.
  
  Thanks for the info (and another reason to keep SQL names in lowercase ;-)).
Be careful!

Some databases convert them to uppercase!
Oracle is a prominent example!

It took me once some ours to locate a bug resulting
from this "feature".


Dieter

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?

2001-01-01 Thread Cees de Groot

Jon Prettyman [EMAIL PROTECTED] said:
If you are using ZSQLMethods and your database returns column names in
one case (UPPER) but you reference them with another (lower) it
appears that instances of SQLAlias get leaked.

Thanks for the info (and another reason to keep SQL names in lowercase ;-)).


-- 
Cees de Groot   http://www.cdegroot.com [EMAIL PROTECTED]
GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD  1986 F303 937F E098 9E8B

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?

2000-12-30 Thread Jon Prettyman

If you are using ZSQLMethods and your database returns column names in
one case (UPPER) but you reference them with another (lower) it
appears that instances of SQLAlias get leaked.

Brian Lloyd has put Jim Fulton's fix in the 2.2.5b release that's up
on the site.

We were seeing thousands of leaked instances of SQLAlias causing our
memory usage to skyrocket.

-jon

[EMAIL PROTECTED] (Cees de Groot) writes:

 Jon Prettyman [EMAIL PROTECTED] said:
 Are you using ZSQLMethods by any chance?
 
 Yes, why? (I think I've fixed /this/ problem, but if ZSQLMethods will
 cause similar behaviour I'd like to be prepared :-))
 
 -- 

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?

2000-12-29 Thread Jon Prettyman

Are you using ZSQLMethods by any chance?

-jon

[EMAIL PROTECTED] (Cees de Groot) writes:

 Our production Zope machine has developed a habit over Christmas where
 it starts eating CPU and RAM (one Python thread on 100% CPU time, and 
 I've seen kernel messages indicating that Zope was killed because it
 hit an out-of-memory condition which means it sat on ~500Mb RAM). 
 
 I'm completely in the dark about what causes this (probably some of
 my own code, I installed some new bits a couple of days ago), but what's
 worse: I really don't know how to set about finding the problematic
 code. Is there a way to find out what code a thread is running?
 
 -- 
 Cees de Groot   http://www.cdegroot.com [EMAIL PROTECTED]
 GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD  1986 F303 937F E098 9E8B
 
 ___
 Zope maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope
 **   No cross posts or HTML encoding!  **
 (Related lists - 
  http://lists.zope.org/mailman/listinfo/zope-announce
  http://lists.zope.org/mailman/listinfo/zope-dev )

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?

2000-12-29 Thread Cees de Groot


[EMAIL PROTECTED] said:
 I've made zope inaccessible two different ways:
 - passing bad data to ZDGChart - sending mail when MailHost is down
 (i.e. the host it is supposed to connect to is not available) 

It seems a bug in my virtual hosting code (a Python method that acts as the 
AccessRule for the root) pushes an empty string on the 
TraversalRequestNameStack. Could that be the cause?
-- 
Cees de Groot   http://www.cdegroot.com [EMAIL PROTECTED]
GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD  1986 F303 937F E098 9E8B



___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?

2000-12-29 Thread Cees de Groot

Jon Prettyman [EMAIL PROTECTED] said:
Are you using ZSQLMethods by any chance?

Yes, why? (I think I've fixed /this/ problem, but if ZSQLMethods will
cause similar behaviour I'd like to be prepared :-))

-- 
Cees de Groot   http://www.cdegroot.com [EMAIL PROTECTED]
GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD  1986 F303 937F E098 9E8B

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?

2000-12-28 Thread Dieter Maurer

Cees de Groot writes:
  ... excessive CPU/memory consumption ...
If you use Zope 2.2.x, you can try to access the debug tools
in the "Control Panel". They will show you, which request
is active (to more easily reproduce and analyse the problem)
and how the reference counts are changed.

Dieter

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?

2000-12-28 Thread Chris McDonough

Turning on detailed logging at startup might help too... read about the -M
argument in the z2.py script and read my Howto regarding "How To Track Down
Hangs" at http://www.zope.org/Members/mcdonc/HowTos/DEBUG-LOG


"Cees de Groot" [EMAIL PROTECTED] wrote in message
92ghr3$jf3$[EMAIL PROTECTED]">news:92ghr3$jf3$[EMAIL PROTECTED]...
 Dieter Maurer [EMAIL PROTECTED] said:
 Cees de Groot writes:
   ... excessive CPU/memory consumption ...
 If you use Zope 2.2.x, you can try to access the debug tools
 in the "Control Panel". They will show you, which request
 is active (to more easily reproduce and analyse the problem)
 and how the reference counts are changed.
 
 The problem is, apparently the thing gets in a really tight loop - one
 threads blocks the rest of Zope, so it is quite hard to use these
 tools (especially not when the whole thing is not - yet - reproducable,
 sometimes it happens in an hour, sometimes after 12 hours; a bit hard
 to keep staring at the screen for 12 hours :-)).

 As a bit of an act of despair, I'm currently logging Zope's CPU usage
 every 5 seconds in a log file, hopefully I can match a rise in CPU time
 with some request in the server logs.


 --
 Cees de Groot   http://www.cdegroot.com [EMAIL PROTECTED]
 GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD  1986 F303 937F E098 9E8B

 ___
 Zope maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope
 **   No cross posts or HTML encoding!  **
 (Related lists -
  http://lists.zope.org/mailman/listinfo/zope-announce
  http://lists.zope.org/mailman/listinfo/zope-dev )




___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?

2000-12-28 Thread Cees de Groot

Dieter Maurer [EMAIL PROTECTED] said:
Cees de Groot writes:
  ... excessive CPU/memory consumption ...
If you use Zope 2.2.x, you can try to access the debug tools
in the "Control Panel". They will show you, which request
is active (to more easily reproduce and analyse the problem)
and how the reference counts are changed.

The problem is, apparently the thing gets in a really tight loop - one
threads blocks the rest of Zope, so it is quite hard to use these
tools (especially not when the whole thing is not - yet - reproducable,
sometimes it happens in an hour, sometimes after 12 hours; a bit hard
to keep staring at the screen for 12 hours :-)).

As a bit of an act of despair, I'm currently logging Zope's CPU usage
every 5 seconds in a log file, hopefully I can match a rise in CPU time
with some request in the server logs. 


-- 
Cees de Groot   http://www.cdegroot.com [EMAIL PROTECTED]
GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD  1986 F303 937F E098 9E8B

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




Re: [Zope] Zope eating CPU/RAM - how do I find the culprit?

2000-12-28 Thread Cees de Groot


[EMAIL PROTECTED] said:
 Turning on detailed logging at startup might help too... read about
 the -M argument in the z2.py script and read my Howto regarding "How
 To Track Down Hangs" at http://www.zope.org/Members/mcdonc/HowTos/
 DEBUG-LOG 

Thanks! That looks promising!
-- 
Cees de Groot   http://www.cdegroot.com [EMAIL PROTECTED]
GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD  1986 F303 937F E098 9E8B



___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )




[Zope] Zope eating CPU/RAM - how do I find the culprit?

2000-12-27 Thread Cees de Groot

Our production Zope machine has developed a habit over Christmas where
it starts eating CPU and RAM (one Python thread on 100% CPU time, and 
I've seen kernel messages indicating that Zope was killed because it
hit an out-of-memory condition which means it sat on ~500Mb RAM). 

I'm completely in the dark about what causes this (probably some of
my own code, I installed some new bits a couple of days ago), but what's
worse: I really don't know how to set about finding the problematic
code. Is there a way to find out what code a thread is running?

-- 
Cees de Groot   http://www.cdegroot.com [EMAIL PROTECTED]
GnuPG 1024D/E0989E8B 0016 F679 F38D 5946 4ECD  1986 F303 937F E098 9E8B

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )