On Mon, Jul 19, 2010 at 9:40 AM, Andreas Gabriel
<gabr...@hrz.uni-marburg.de> wrote:
> Hello,
> we are hosting a big ZEO based zope site and encountered a problem with a
> limited size of file descriptors (FDs) of the ZEO server.
> Scenario
>  36 ZEO clients (each 2 threads)
>  1  ZEO server  (serving 10 storages)

I advise against serving more than one storage per server.

> It was not possible to connect all ZEO clients to the ZEO server.
> After a short amount of time following events occur in the event.log
> of the ZEO clients
> [snip]
> 2010-02-08T14:03:25 PROBLEM(100) zrpc:21615 CW: error connecting to
> ('zeo-server.dummy.de', 10000): ECONNREFUSED
> [snip]
> and simultaneously the ZEO server hangs and the whole site goes down.
> Unfortunately, there was no hint in the ZEO server logs. After 'Googling'
> we found the following hint
>  http://comments.gmane.org/gmane.comp.web.zope.plone.user/101892
> that each zeo client connection is consuming three file descriptors at the ZEO
> server side. It was possible to calculate the theoretical required number of 
> FDs
> with this info
>   75 (base) + 36 (zeo-clients) x 10 (storages) = 1155
> We tried to open as many connections as possible to the ZEO server with a 
> simple
> script (see attachment) and counted the number of open FDs of the ZEO server 
> using "lsof".
> The result was that the ZEO server hangs at 1025 open FDs. Therefore, we 
> assumed that
> the OS (here Linux) limits the available number of FDs to 1024 by 
> configuration.
> Using "ulimit" (hard/soft) we increased the number of allowed open FDs to 
> 2048.
> However, there was no chance to open more than 329 (instead of 360) 
> connections
> (=1025 FDs) to the ZEO server :(
> After looking at the sources, ZEO server uses the asyncore library to manage 
> the
> incoming connections. After *intensive* 'Googling' we have to notice that 
> python's
> asyncore library has a hard compiled in size limit of open FDs (namely 1024). 
> The
> limit is defined as macro __FD_SETSIZE in the header file of the libc6 library
>    /usr/include/bits/typesizes.h
> Therefore, it was unfortunately necessary to change the limit in the header 
> file
> to
>  #define __FD_SETSIZE 2048
> and to re-compile python's sources to overcome the problem. However, our ZEO 
> scenario
> now works with the re-compiled python interpreter :)

A simpler solution is to run each storage in a separate process.

(In ZODB 3.9, only one fd is needed per client.  ZODB 3.10 goes back
 to requiring 3.)


Jim Fulton
For more information about ZODB, see the ZODB Wiki:

ZODB-Dev mailing list  -  ZODB-Dev@zope.org

Reply via email to