Re: [Zope-dev] Memory 2.1.4-2.1.6 a.k.a. how to get objects out of the cache

2000-05-30 Thread Jason Spisak

Bill Anderson:

 Jason Spisak wrote:
 
 [...]
 
  That's funny that I am experienceing the same symptoms as you, but my
  installation is totally different. I'm not using GUF, or any DA's, and I'm
  running on RH 6.1.  We really need to figure this out as a community.  I am
  willing to try just about anything short of shipping my machine somewhere
  off premisis. ;)
 
 
 Ok, let's dig up some more info ( or perhaps I missed it ).
 
 Server setup, as in pcgi/Zserver/Apache w/proxypass, etc?

Nothing but straight ZServer/Medusa.

 Compiled on the machine, or binary distribution?
 

 Zope 2.1.1 binary distro. Not sure how it got to be 2.1.4/2.1.6, might
have been the other person having problems, or my mistake.  But the changes
in 2.1.4 and 2.1.6 aren't anything in the Object publisher or Medusa so I
don't think it makes a difference.


 If source, did you start with a clean tree, or did use use the diff
 upgrade, or did you use a new 2.1.6 tarball, and drop it on top of the
 2.1.4 install?
 
 Bill

RH 6.1
2.2.12smp kernel
Hardware RAID
dual 350mhz Xenon 
1 gig ram

Vanilla Zope with no DA's, no Apache.


BTW, Matthew gave me some advice last Friday to use the setcheckinterval
line in z2.py and the performance gains are quite impressive.  Anyone who
is running smp should try it.


All my best,

Jason Spisak
CIO
HireTechs.com
6151 West Century Boulevard
Suite 900
Los Angeles, CA 90045
P. 310.665.3444
F. 310.665.3544

Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email
address may not be added to any commercial mail list with out my
permission.  Violation of my privacy with advertising or SPAM will
result in a suit for a MINIMUM of $500 damages/incident, $1500 for
repeats.

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )




Re: [Zope-dev] Memory 2.1.4-2.1.6 a.k.a. how to get objects out of the cache

2000-05-30 Thread Bill Anderson

Jason Spisak wrote:
 
 Bill Anderson:
 
  Jason Spisak wrote:
 
  [...]
 
   That's funny that I am experienceing the same symptoms as you, but my
   installation is totally different. I'm not using GUF, or any DA's, and I'm
   running on RH 6.1.  We really need to figure this out as a community.  I am
   willing to try just about anything short of shipping my machine somewhere
   off premisis. ;)
 
 
  Ok, let's dig up some more info ( or perhaps I missed it ).
 
  Server setup, as in pcgi/Zserver/Apache w/proxypass, etc?
 
 Nothing but straight ZServer/Medusa.

Cool, eliminates suspects.
 
  Compiled on the machine, or binary distribution?
 
 
  Zope 2.1.1 binary distro. Not sure how it got to be 2.1.4/2.1.6, might
 have been the other person having problems, or my mistake.  But the changes
 in 2.1.4 and 2.1.6 aren't anything in the Object publisher or Medusa so I
 don't think it makes a difference.

Have you considered trying a source-compile setup? There may have been
something in the original binary that gets triggered when dynamically
linking (does it do that? I haven't run a binary Zope distribution) to
the libraries on your machine.

I have witnessed this with X.

 RH 6.1
 2.2.12smp kernel
 Hardware RAID
 dual 350mhz Xenon
 1 gig ram
 
 Vanilla Zope with no DA's, no Apache.
 
 BTW, Matthew gave me some advice last Friday to use the setcheckinterval
 line in z2.py and the performance gains are quite impressive.  Anyone who
 is running smp should try it.

It is impressive. I love seing Zope serving static files in the 70rps
range.

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )




Re: [Zope-dev] Memory 2.1.4-2.1.6 a.k.a. how to get objects out of the cache

2000-05-29 Thread Eric Sattler

Shane Hathaway wrote:
 
 Stuart 'Zen' Bishop wrote:
  /me hops on the 'real garbage collection for Python' bandwagon
 
 Have you looked at this?
Yes, I actually built, installed and enabled it.  It doesn't seem
to have a major impact on my problems at least...neither does Zen's
latest set of fixes.  I enabled Neil Schemenauer's fixes, rebuilt 
python, and then tried tracking through chunks of the CPickleCache
code.  There did not seem to be any hanging python references.  I 
believe the problem to be circular zope references on a higher level.

I could be wrong though.  The gc code for cPickleCache.c is not trivial,
and I believe there is something wrong there, but have not proved it
yet.  This is the code that determines when things are removed from the
zope cache.

The wierd thing is that in my case, I am not really writing anything
to the Zope db.  Everything is stored externally in a postgres database.
I am using GUF, and SQLSession though.  

Could it be that connection objects are not being cleaned up properly?
As far as I understand it, connection objects are containers for zope
objects
for the duration of that request.

At least in the case of postgres 6.5.3 there was some connection memory
leaks, so I did try postgres 7.0, it solved some of postgres's problem
but not zope's.

sorry for rambling, just trying to get all of the tidbits that I know
out there, maybe someone will see something.

eric.


 
 http://www.enme.ucalgary.ca/~nascheme/python/gc.html
 
 I'd like to know how successful this project is.  I wonder whether it
 would have an issue with ExtensionClass instances (one of the
 foundations of Zope).
CPickleCache
 
 Shane
 
 ___
 Zope-Dev maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists -
  http://lists.zope.org/mailman/listinfo/zope-announce
  http://lists.zope.org/mailman/listinfo/zope )

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )




Re: [Zope-dev] Memory 2.1.4-2.1.6 a.k.a. how to get objects out of the cache

2000-05-28 Thread Eric Sattler

All,

I also am having severe problems with memory creep.  Our problem
exhibits itself by quickly using gobs of memory and requiring a
zope restart after about a day and a half at about 100M resident.  

First our configuration:

Core Components:

FreeBSD 4.0/FreeBSD 3.4 - on separate machines of course
Postgres 6.5.3/Postgres 7.0(fixes some memory leaks)
Python 1.5.2
Zope 2.1.4/2.1.6 have tried both
pcgi with ZServer
Apache w/mod_ssl 1.3.9 and 1.3.12

Products:

ZPySQLDA
SQLSession
GenericUserFolder

I have looked at a lot of different things.  This includes:

Installing the gc(garbage collection) python package 
   - Neil Schemenauer's patches
Installing Postgres 7.0(fixes some connection memory leaks)
Tracking the cache cleanup inside zope
Tracking thread locking inside DB.py, Transaction.py

Results:

gc python didn't help alleviate the problem
Postgres 7.0 instead of 6.5.3 didn't help the problem

I do see GenericUserFolder and SQLSession objects with the
Control_Panel_Debug screen, and they do not seem to go away.  I wrote
a simple python script to do nothing more than authenticate (log in)
using the GenericUserFolder method docLogin.  The memory usage quickly
grows out of control.  After waiting 15 minutes(my cookie timeout),
no decrease in memory usage.  The objects are still in the cache also. 

Restarting zope clears everything up, and starts out nice and clean
again.
We have a cron job doing this right now.

Questions:

1.  How does a user's connection resources etc. get cleaned up
by Zope after a cookie timeout?  Do I need to do this myself?

2.  How does a genericuserfolder's set of objects used by
an   
authenticated user get cleaned up.  I can't seem to make those
objects get reclaimed by the system, even when the user logs
off?

3.  I am not explicitly removing SQLSession objects.  Will my usage
counts for these objects remain  1, thereby never allowing them
to get cleaned up, and thereby keeping my usage counts for 
genericuserfolder's  1?

4.  Since GenericUserFolder inherits from Folder objects, is it 
possible that the reason my memory grows so fast is that Zope
does not release the resources used by a user properly, during
an abrupt disconnect?

I will assist/help trying different things to try to solve this problem!

thanks.

eric.

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )




Re: [Zope-dev] Memory 2.1.4-2.1.6 a.k.a. how to get objects out of the cache

2000-05-28 Thread Bill Anderson

Eric Sattler wrote:
 
 All,
 
 I also am having severe problems with memory creep.  Our problem
 exhibits itself by quickly using gobs of memory and requiring a
 zope restart after about a day and a half at about 100M resident.

[...]

 I do see GenericUserFolder and SQLSession objects with the
 Control_Panel_Debug screen, and they do not seem to go away.  I wrote
 a simple python script to do nothing more than authenticate (log in)
 using the GenericUserFolder method docLogin.  The memory usage quickly
 grows out of control.  After waiting 15 minutes(my cookie timeout),
 no decrease in memory usage.  The objects are still in the cache also.
 
 Restarting zope clears everything up, and starts out nice and clean
 again.
 We have a cron job doing this right now.


This seems to dovetail with the reports of SQLSession being a memory
leak source.
Could you drop the SQLSession, and repeat the test?

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )




Re: [Zope-dev] Memory 2.1.4-2.1.6 a.k.a. how to get objects out of the cache

2000-05-26 Thread Bill Anderson

Jason Spisak wrote:

[...]

 That's funny that I am experienceing the same symptoms as you, but my
 installation is totally different. I'm not using GUF, or any DA's, and I'm
 running on RH 6.1.  We really need to figure this out as a community.  I am
 willing to try just about anything short of shipping my machine somewhere
 off premisis. ;)


Ok, let's dig up some more info ( or perhaps I missed it ).

Server setup, as in pcgi/Zserver/Apache w/proxypass, etc?
Compiled on the machine, or binary distribution?

If source, did you start with a clean tree, or did use use the diff
upgrade, or did you use a new 2.1.6 tarball, and drop it on top of the
2.1.4 install?

Bill

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )