Re: [Zope-dev] Very severe memory leak
On 08/25/2003 05:12 PM, Leonardo Rochael Almeida wrote:
But maybe this means that the leak is not related to the DateTime
refcounts. It's just that the fast and continual increase in DateTime
objects is really bugging me. BTW, what is the usual DateTime refcount
in heavy ZCatalog sites you guys run?
We try to keep it near 0. :-) DateTimes are full of nonsense. Zope's
DateTime troubles drove the creation of Python 2.3's new datetime
module. From what I can tell, the new datetimes are excellent.
Let me restate something important, because I forget about it myself
sometimes when I'm thinking about this problem: *upgrading to 2.6.1 made
the situation definetly worse*. The site became lot faster, but the
memory situation became intolerable, so I believe there is something
introduced between 2.5.1 and 2.6.1 that is causing our problems. Is
there a fix in 2.6.2 for some leak introduced *in* the 2.6 series?
I think there are, actually, but AFAIK they were obscure and rare.
Another thing, we have already estabilished that clearing the caches
resets the DateTime refcount, so the DateTime must be anchored to the
caches somehow. So, if the DateTime refcounts are constantly in the
50k-100k range, how come the cache object count for all threads is below
5k?
If the DateTimes are in catalog metadata or indexes, this would make sense.
If they're indirectly attached, it means that a lot of DateTimes are
anchored in just a few objects, so no matter how much I reduce the
target cache size, it might never be enough to throw out the
DateTimes...
They are probably in the ZCatalog. Remove all metadata fields you can.
If you have index dates, use DateTimeIndexes rather than the standard
indexes.
I find it helpful to simply visit the information ZCatalog is storing
for just one cataloged object. In the catalog contents view, click one
object. Look at each field and consider how ZCatalog might misuse that
field for other objects.
hmm, any chance of this remote console showing up somewhere? :-)
I've attached what I threw together. It Works For Me. ;-) I put it in
Zope's Extensions directory then made an external method to call the
listenlocal function. It sets up a TCP listener bound only to
localhost, so you have to telnet to port 8765 locally. No readline
support. No method is provided for shutting down the listener (except
restarting Zope). To exit the session, push ctrl-D then enter.
I might use it to navigate the thread object cache.
It's unfortunate you have to go so deep, but that sounds like a good plan.
Shane
import socket
from code import InteractiveConsole
from types import IntType, StringType
from thread import start_new_thread, get_ident
import sys
import time
class ThreadedObjectProxy:
Proxy to different objects based which thread invoked it.
def __init__(self, default):
self._default = default
self._alts = {}
def setAlternative(self, o):
self._alts[get_ident()] = o
def delAlternative(self):
try: del self._alts[get_ident()]
except KeyError: pass
def __getattr__(self, name):
ob = self._alts.get(get_ident(), self._default)
return getattr(ob, name)
class RemoteConsole(InteractiveConsole):
def __init__(self, sock, file, filename=None, locals=None):
if filename is None:
filename = str(file)
self.sock = sock
self.file = file
InteractiveConsole.__init__(self, locals=locals, filename=filename)
def raw_input(self, prompt=''):
if prompt:
self.file.write(prompt)
s = self.file.readline().rstrip()
if s == '\x04': # Ctrl-D
raise EOFError
return s
def interactAndClose(self):
sys.stdout.setAlternative(self.file)
sys.stderr.setAlternative(self.file)
sys.stdin.setAlternative(self.file)
try:
try:
self.interact()
except EOFError:
pass
finally:
sys.stdout.delAlternative()
sys.stderr.delAlternative()
sys.stdin.delAlternative()
self.file.close()
self.sock.close()
def setupStreams():
if not hasattr(sys.stdout, 'setAlternative'):
sys.stdout = ThreadedObjectProxy(sys.stdout)
if not hasattr(sys.stderr, 'setAlternative'):
sys.stderr = ThreadedObjectProxy(sys.stderr)
if not hasattr(sys.stdin, 'setAlternative'):
sys.stdin = ThreadedObjectProxy(sys.stdin)
def accept_connections(s):
while 1:
cs, addr = s.accept()
f = cs.makefile('w+', 0)
i = RemoteConsole(cs, f)
start_new_thread(i.interactAndClose, ())
def listen(addr, locals=None):
setupStreams()
if isinstance(addr, StringType):
t = socket.AF_UNIX
elif isinstance(addr, IntType):
t = socket.AF_INET
addr = ('', addr)
else:
t = socket.AF_INET
s = socket.socket(t, socket.SOCK_STREAM)
[Zope-dev] zope.org down....
Sorry for the offtopic/crosspost but, www.zope.org is down, and with a rather ugly error message: ugly_message ERROR The requested URL could not be retrieved While trying to retrieve the URL: http://12.155.117.33:8082/VirtualHostBase/http/zope.org:80/zopeorg/VirtualHostRoot/ The following error was encountered: Connection Failed The system returned: (111) Connection refused The remote host or network may be down. Please try the request again. Your cache administrator is [EMAIL PROTECTED] /ugly_message hope someone can do something ;) Romain Slootmaekers ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Very severe memory leak
I have got about 7 DateTime refs on that one CMF site at the moment and can arbitrarily increase this number by serving out pages ;-). They don't stick though, but go back to a stable 4. No cause for alarm here. Did you rebuild the catalog (-indexes) after the upgrade? New Zope versions usually come with important BTree fixes ;-). Wild guess, but do you by any chance run exUserFolder? hth, Stefan --On Montag, 25. August 2003 18:12 -0300 Leonardo Rochael Almeida [EMAIL PROTECTED] wrote: But maybe this means that the leak is not related to the DateTime refcounts. It's just that the fast and continual increase in DateTime objects is really bugging me. BTW, what is the usual DateTime refcount in heavy ZCatalog sites you guys run? -- The time has come to start talking about whether the emperor is as well dressed as we are supposed to think he is. /Pete McBreen/ ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Very severe memory leak
Paul Winkler wrote: On Mon, Aug 25, 2003 at 10:35:58PM -0400, Shane Hathaway wrote: They are probably in the ZCatalog. Remove all metadata fields you can. If you have index dates, use DateTimeIndexes rather than the standard indexes. Hmm. ZCatalog provides a DateIndex, is this what you mean? Googling for DateTimeIndex zope turns up exactly two results, in German which I can't read. (the google-supplied translations are pretty much gibberish.) Yes, I meant DateIndex. Hmmm again: I notice that out-of-the-box, CMF Default's portal_catalog uses the following: DateFieldIndex created FieldIndex effective FieldIndex expires FieldIndex modifiedFieldIndex Why are these FieldIndex rather than DateTime index? No one has corrected them, probably out of fear that something might break. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Why not ZODB 3.1.3 with Zope 2.6.2?
I downloaded the hidden 2.6.2b6 release (from http://zope.org/Products/Zope/2.6.2b6/Zope/view) and had a look in lib/python/ZODB/__init__.py, which gives its version as ZODB 3.1.2. Is there any reason it's not ZODB 3.1.3? Maybe just an oversight? I could easily check any ZODB-3.1.3 changes to the Zope-2.6 branch, unless somebody knows a reason not to do so. Also, why doesn't ZODB/Tools come with zope? There's some really handy stuff in there, some of which gets mentioned on the zope list from time to time; it would be nice if newbies could get hold of it easily. On a related note, NEWS.txt in ZODB 3.1.3 mentions that there is a new Tools/README.txt ... but it's not actually in the tarball :-) -- Paul Winkler http://www.slinkp.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Very severe memory leak
On Tue, 2003-08-26 at 12:05, Stefan H. Holek wrote: [...] Did you rebuild the catalog (-indexes) after the upgrade? New Zope versions usually come with important BTree fixes ;-). If by rebuilding you mean clicking on the Update button, then yes, as part of changing the DateTime FieldIndexes to DateIndexes. If you meant something else, then no. Wild guess, but do you by any chance run exUserFolder? As a matter of fact, I do. Is there a known leak problem with exUserFolder? -- Ideas don't stay in some minds very long because they don't like solitary confinement. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
