On Mon, Jul 19, 2010 at 9:40 AM, Andreas Gabriel
> we are hosting a big ZEO based zope site and encountered a problem with a
> limited size of file descriptors (FDs) of the ZEO server.
> 36 ZEO clients (each 2 threads)
> 1 ZEO server (serving 10 storages)
I advise against serving more than one storage per server.
> It was not possible to connect all ZEO clients to the ZEO server.
> After a short amount of time following events occur in the event.log
> of the ZEO clients
> 2010-02-08T14:03:25 PROBLEM(100) zrpc:21615 CW: error connecting to
> ('zeo-server.dummy.de', 10000): ECONNREFUSED
> and simultaneously the ZEO server hangs and the whole site goes down.
> Unfortunately, there was no hint in the ZEO server logs. After 'Googling'
> we found the following hint
> that each zeo client connection is consuming three file descriptors at the ZEO
> server side. It was possible to calculate the theoretical required number of
> with this info
> 75 (base) + 36 (zeo-clients) x 10 (storages) = 1155
> We tried to open as many connections as possible to the ZEO server with a
> script (see attachment) and counted the number of open FDs of the ZEO server
> using "lsof".
> The result was that the ZEO server hangs at 1025 open FDs. Therefore, we
> assumed that
> the OS (here Linux) limits the available number of FDs to 1024 by
> Using "ulimit" (hard/soft) we increased the number of allowed open FDs to
> However, there was no chance to open more than 329 (instead of 360)
> (=1025 FDs) to the ZEO server :(
> After looking at the sources, ZEO server uses the asyncore library to manage
> incoming connections. After *intensive* 'Googling' we have to notice that
> asyncore library has a hard compiled in size limit of open FDs (namely 1024).
> limit is defined as macro __FD_SETSIZE in the header file of the libc6 library
> Therefore, it was unfortunately necessary to change the limit in the header
> #define __FD_SETSIZE 2048
> and to re-compile python's sources to overcome the problem. However, our ZEO
> now works with the re-compiled python interpreter :)
A simpler solution is to run each storage in a separate process.
(In ZODB 3.9, only one fd is needed per client. ZODB 3.10 goes back
to requiring 3.)
For more information about ZODB, see the ZODB Wiki:
ZODB-Dev mailing list - ZODB-Dev@zope.org