Having a single ZEO server handling more than one or two storages is not usually a good idea. ZEO does not handle each storage in a separate thread, so you're underusing multiple CPUs if you have them.
Multiple ZEO servers would also spread the open sockets among them. There could still be a leak, but it'd take longer to exhaust file descriptors. On Wed, Oct 17, 2012 at 11:29 PM, Tim Godfrey <t...@obsidian.com.au> wrote: > Hi all > > We have a customer on Plone 4.1 and recently we started to see a great > increase in the number of open file descriptors the Zeoserver process uses. > > We have a pretty big system distributed over many servers as the Plone > system gets a lot of requests. Currently we are running a Zeoserver with 30 > storages and have 16 Zope clients that connect to it. > > I've had Zeoserver in debug mode to try and get some more insight into the > problem and I'm seeing a lot of these errors: > > Exception in thread Thread-575: > Traceback (most recent call last): > File "/opt/informa/Python-2.6/lib/python2.6/threading.py", line 532, in > __bootstrap_inner > self.run() > File "/opt/informa/Python-2.6/lib/python2.6/threading.py", line 484, in > run > self.__target(*self.__args, **self.__kwargs) > File > "/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/zrpc/connection.py", > line 661, in server_loop > asyncore.poll(30.0, map) > File "/opt/informa/Python-2.6/lib/python2.6/asyncore.py", line 140, in > poll > r, w, e = select.select(r, w, e, timeout) > ValueError: filedescriptor out of range in select() > > Exception in thread Thread-576: > Traceback (most recent call last): > File "/opt/informa/Python-2.6/lib/python2.6/threading.py", line 532, in > __bootstrap_inner > self.run() > File "/opt/informa/Python-2.6/lib/python2.6/threading.py", line 484, in > run > self.__target(*self.__args, **self.__kwargs) > File > "/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/zrpc/connection.py", > line 661, in server_loop > asyncore.poll(30.0, map) > File "/opt/informa/Python-2.6/lib/python2.6/asyncore.py", line 140, in > poll > r, w, e = select.select(r, w, e, timeout) > ValueError: filedescriptor out of range in select() > > There's also a lot of these lines in zeoserver.log: > > 2012-10-18T12:42:08 new connection ('10.221.0.144', 34919): > <ManagedServerConnection ('10.221.0.144', 34919)> > 2012-10-18T12:42:08 new connection ('10.221.0.64', 41539): > <ManagedServerConnection ('10.221.0.64', 41539)> > > Each time a new connection is logged further open file descriptors are used. > > There are also these errors in the Zope clients: > > 2012-10-18T13:20:01 ERROR ZEO.zrpc (7746) CW: error in testConnection > (('10.221.0.247', 8101)) > Traceback (most recent call last): > File > "/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/zrpc/client.py", > line 595, in test_connection > self.preferred = self.client.testConnection(self.conn) > File > "/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/ClientStorage.py", > line 565, in testConnection > stub = self.StorageServerStubClass(conn) > File > "/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/ServerStub.py", > line 386, in stub > raise ValueError("Timeout waiting for protocol handshake") > ValueError: Timeout waiting for protocol handshake > > At time of writing the Zeoserver process is using 1604 file descriptors but > that number will continue to grow. I suppose I'm wondering why it's always > creating new fds and not closing old ones. Also whether there are changes I > can make to my configuration to stop these errors and growing number of file > descriptors. > > If Zeoserver is allowed to keep running it will eventually hit the operating > system limit of open fds and start raising "Too Many Open File" Exceptions > which causes havoc with transactions are I'm sure you would know. > > Is anyone able to offer some advice? I'm more than happy to provide more > information if required. > > Thanks, > > Tim > > > _______________________________________________ > For more information about ZODB, see http://zodb.org/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > https://mail.zope.org/mailman/listinfo/zodb-dev > _______________________________________________ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev