Tim Peters wrote at 2005-12-30 14:51 -0500:
[Dieter Maurer]
They did not tell us about their application. But, Zope's database
adapters work like this. They use the ZODB cache (and its pool) as an RDB
connection pool. Thus, if the ZODB caches are not released, the RDB
connections won't.
I believe you, but I'm not sure what to make of it; for example,
1. The OP is the only person who has this problem?
2. Other people have this problem too, and don't know what to do about
it either, but never complain?
I expect (but the original poster may step in) that the problem
could occur in Zope only when the number of threads exceeds
the pool size (or additional connections are used in an application
specific way) as otherwise, there are no dropped connections.
Because formerly, it made no sense to have more worker threads than
that given by the pool size, this situation is likely to occur
rarely.
...
If not, you may have better luck on the zope-db list (which is
devoted to using other databases with Zope):
The problem is not with the RDB but with the ZODB connections that are
magically not garbage collected. He will probably not find help on
zope-db.
That suggestion was based on a guess that #3 (above) is most likely. Of
course I don't know, but #1 and #2 seem unlikely on the face of it. If
other people using RDB don't have this problem, then zope-db is the right
place to ask how they manage to avoid it.
If the poster has no evidence that the ZODB connections are definitely kept
but just see that the relational database connections remain open,
the reason might indeed be at a completely different place:
There are some reports on zope-db about DAs leaking relational
database connections.
The problem is not related to ZODB connection handling.
Instead, the relational database connection is kept open
even if the DA object was invalidated (and especially cleared).
We observed such behaviour with Zope 2.7/ZODB 3.2 and ZPsycopgDA.
When we used resetCaches (which should in principle release
the old caches (I also added an minimizeCache because
the cyclic gc does not work with Zope 2.7's ExtensionClass objects)),
a new set of Postgres connections was opened without
the old ones being closed.
We worked around this problem by:
* avoiding to use resetCaches
* restarting Zope once a weak to get rid of
stale Postgres connections
...
OTOH, if that's not
what's going on here, I'd expect to have heard about this here more than
once in the last 5 years ;-)
The older ZODB code (ZODB 3.2 and before) was seriously flawed
with respect to cache release handling.
Fortunately, it was very rare that caches need to be released.
I found the flaws because we used temporary connections extensively
and, of course, their caches need to go away with the temporary
connection. I had a hard fight to get rid of the memory leaks
induced by those flaws.
Now (ZODB 3.4 and above) caches might get released more often.
True, the garbage collector now has a chance to collect cycles
including ExtensionClass objects, but it is very easy to
defeat the GC -- an object with a __del__ method is sufficient.
Perhaps because the OP is unique in allowing
hundreds (or thousands -- whatever) of Connections to be active
simultaneously? Don't know.
He did not say that.
If his hypothesis is indeed true and some connections exceeding
the pool size are kept indefinitely, then already slightly
exceeding the pool size may lead to an unbound number of connections
provided the exceedance occurs frequently enough.
I suggested before that forcing calls to gc.collect() would give more
evidence. If that doesn't help, then it's most likely that the application
is keeping Connections alive. Since I'm not familiar with the RDB code, I
suppose it's even possible that such code uses __del__ methods, and creates
cycles of its own, that prevent cyclic gc from reclaiming them. In that
case, there are serious problems in that code.
There is also another gc attribute holding the garbage cycles
not released. The poster may examine them to check whether they
contain indeed ZODB connection objects.
--
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
