we are hosting a big ZEO based zope site and encountered a problem with a
limited size of file descriptors (FDs) of the ZEO server.
36 ZEO clients (each 2 threads)
1 ZEO server (serving 10 storages)
It was not possible to connect all ZEO clients to the ZEO server.
After a short amount of time following events occur in the event.log
of the ZEO clients
2010-02-08T14:03:25 PROBLEM(100) zrpc:21615 CW: error connecting to
('zeo-server.dummy.de', 10000): ECONNREFUSED
and simultaneously the ZEO server hangs and the whole site goes down.
Unfortunately, there was no hint in the ZEO server logs. After 'Googling'
we found the following hint
that each zeo client connection is consuming three file descriptors at the ZEO
server side. It was possible to calculate the theoretical required number of FDs
with this info
75 (base) + 36 (zeo-clients) x 10 (storages) = 1155
We tried to open as many connections as possible to the ZEO server with a simple
script (see attachment) and counted the number of open FDs of the ZEO server
The result was that the ZEO server hangs at 1025 open FDs. Therefore, we
the OS (here Linux) limits the available number of FDs to 1024 by configuration.
Using "ulimit" (hard/soft) we increased the number of allowed open FDs to 2048.
However, there was no chance to open more than 329 (instead of 360) connections
(=1025 FDs) to the ZEO server :(
After looking at the sources, ZEO server uses the asyncore library to manage the
incoming connections. After *intensive* 'Googling' we have to notice that
asyncore library has a hard compiled in size limit of open FDs (namely 1024).
limit is defined as macro __FD_SETSIZE in the header file of the libc6 library
Therefore, it was unfortunately necessary to change the limit in the header file
#define __FD_SETSIZE 2048
and to re-compile python's sources to overcome the problem. However, our ZEO
now works with the re-compiled python interpreter :)
I hope you will find this information useful.
Dr. Andreas Gabriel, Hochschulrechenzentrum, http://www.uni-marburg.de/hrz
Hans-Meerwein-Str., 35032 Marburg, fon +49 (0)6421 28-23560 fax 28-26994
-------------------- Philipps-Universitaet Marburg -----------------------
"""Connect to a ZEO server and check for maximal connections.
Usage: zeo-check-max-conections.py [options]
-p port -- port to connect to
-h host -- host to connect to (default is current host)
-U path -- Unix-domain socket to connect to
-S name -- comma separated list of storage names (default is '1')
-c connections -- simultaneous connections
You must specify either -p and -h or -U.
from ZEO.ClientStorage import ClientStorage
def multiConnect(addr, storages, connections):
for s in storages:
for i in range(0,connections):
key = '%s-%s' % (s,i)
print 'connecting storage %s' % key
cs[key] = ClientStorage(addr, storage=s, wait=1, read_only=0)
print '%s. connection established' % (len(cs))
# release connections after 10 seconds
for s in cs.keys():
print " ".join(sys.argv)
host = None
port = None
unix = None
storages = ['1']
connections = 1
opts, args = getopt.getopt(sys.argv[1:], 'p:h:U:S:c:')
for o, a in opts:
if o == '-p':
port = int(a)
elif o == '-h':
host = a
elif o == '-U':
unix = a
elif o == '-S':
storages = a.split(',')
elif o == '-c':
connections = int(a)
except Exception, err:
if unix is not None:
addr = unix
if host is None:
host = socket.gethostname()
if port is None:
addr = host, port
multiConnect(addr, storages, connections)
if __name__ == "__main__":
except Exception, err:
For more information about ZODB, see the ZODB Wiki:
ZODB-Dev mailing list - ZODB-Dev@zope.org