Re: [ZODB-Dev] ZeoServer, multiple storages and open file handles
On Wed, Oct 24, 2012 at 12:33 AM, Tim Godfrey t...@obsidian.com.au wrote: Hi Jim Do you have any idea as to why people recommend against many storages under a single Zeo? Seriously? Go back and read the thread. Also can increasing the invalidation-queue-size help this if there is memory to spare on the machine? Invalidation queue size has nothing to do with this. Jim Tim On 23 October 2012 00:53, Jim Fulton j...@zope.com wrote: On Sun, Oct 21, 2012 at 8:48 PM, Tim Godfrey t...@obsidian.com.au wrote: Hi Jon Thanks for your response. Is that something that has been done in a later version of Zeoserver than mine (ZODB3-3.10.3)? No. Every time I've tried to switch to poll in a asyncore, I've had problems. It this your recommended action for the issue I'm having or are there still some configuration changes I can make? Many people have recommended not hosting many storages in a single process. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm -- Tim Godfrey Obsidian Consulting Group P: +61 3 9355 7844 F: +61 3 9350 4097 E: t...@obsidian.com.au W: http://www.obsidian.com.au/ -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZeoServer, multiple storages and open file handles
On 25 October 2012 00:13, Jim Fulton j...@zope.com wrote: On Wed, Oct 24, 2012 at 12:33 AM, Tim Godfrey t...@obsidian.com.au wrote: Hi Jim Do you have any idea as to why people recommend against many storages under a single Zeo? Seriously? Go back and read the thread. Seriously mate. The why is very vague. Also can increasing the invalidation-queue-size help this if there is memory to spare on the machine? Invalidation queue size has nothing to do with this. Once again very well explained thanks. -- Tim Godfrey Obsidian Consulting Group P: +61 3 9355 7844 F: +61 3 9350 4097 E: t...@obsidian.com.au W: http://www.obsidian.com.au/ ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZeoServer, multiple storages and open file handles
On 2012-10-25 00:11:40 +, Tim Godfrey said: On 25 October 2012 00:13, Jim Fulton j...@zope.com wrote: On Wed, Oct 24, 2012 at 12:33 AM, Tim Godfrey t...@obsidian.com.au wrote: Hi Jim Do you have any idea as to why people recommend against many storages under a single Zeo? Seriously? Go back and read the thread. Seriously mate. The why is very vague. There is nothing vague about the first reply to your question: ZEO does not handle each storage in a separate thread, so you're underusing multiple CPUs if you have them. Multiple ZEO servers would also spread the open sockets among them. There could still be a leak, but it'd take longer to exhaust file descriptors. Spread the storages across multiple ZEOs. Also can increasing the invalidation-queue-size help this if there is memory to spare on the machine? Invalidation queue size has nothing to do with this. Once again very well explained thanks. This is free support. No one is really obligated to help, so please be more patient and appreciative of those that try to do so. Cheers -- Tim Godfrey Obsidian Consulting Group P: +61 3 9355 7844 F: +61 3 9350 4097 E: t...@obsidian.com.au W: http://www.obsidian.com.au/ ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev -- Alex Clark · https://www.gittip.com/aclark4life/___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZeoServer, multiple storages and open file handles
Patience goes both ways. Thanks for your help Tim On 25 October 2012 12:33, Alex Clark acl...@aclark.net wrote: ** On 2012-10-25 00:11:40 +, Tim Godfrey said: On 25 October 2012 00:13, Jim Fulton j...@zope.com wrote: On Wed, Oct 24, 2012 at 12:33 AM, Tim Godfrey t...@obsidian.com.au wrote: Hi Jim Do you have any idea as to why people recommend against many storages under a single Zeo? Seriously? Go back and read the thread. Seriously mate. The why is very vague. There is nothing vague about the first reply to your question: ZEO does not handle each storage in a separate thread, so you're underusing multiple CPUs if you have them. Multiple ZEO servers would also spread the open sockets among them. There could still be a leak, but it'd take longer to exhaust file descriptors. Spread the storages across multiple ZEOs. Also can increasing the invalidation-queue-size help this if there is memory to spare on the machine? Invalidation queue size has nothing to do with this. Once again very well explained thanks. This is free support. No one is really obligated to help, so please be more patient and appreciative of those that try to do so. Cheers -- Tim Godfrey Obsidian Consulting Group P: +61 3 9355 7844 F: +61 3 9350 4097 E: t...@obsidian.com.au W: http://www.obsidian.com.au/ ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev -- Alex Clark · https://www.gittip.com/aclark4life/ ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev -- Tim Godfrey Obsidian Consulting Group P: +61 3 9355 7844 F: +61 3 9350 4097 E: t...@obsidian.com.au W: http://www.obsidian.com.au/ ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZeoServer, multiple storages and open file handles
On Wed, Oct 24, 2012 at 12:33 AM, Tim Godfrey t...@obsidian.com.au wrote: Hi Jim Do you have any idea as to why people recommend against many storages under a single Zeo? Seriously? Go back and read the thread. Seriously mate. The why is very vague. There is nothing vague about the first reply to your question: ZEO does not handle each storage in a separate thread, so you're underusing multiple CPUs if you have them. Multiple ZEO servers would also spread the open sockets among them. There could still be a leak, but it'd take longer to exhaust file descriptors. Yes but this is not really strictly accurate. Because it is primarily IO bound, it is certainly possible to do more than 1 CPU worth of work (which Jim notes in a followup email). A useful thread that talks through some of the issues (as they were in 2006, anyway): http://www.gossamer-threads.com/lists/zope/users/201868?do=post_view_threaded#201868 From my point of view, the biggest reason that I have seen *in practice* for splitting off storages into their own separate ZEO was the issue noted here: http://www.gossamer-threads.com/lists/zope/users/201929?do=post_view_threaded#201929 Where Dieter talks about the vote phase on large transactions preventing other requests from running. I have seen this happen in practice (with 2GB transactions) and other requests were, without doubt, prevented from being serviced during this phase. I hope this helps, regards, Darryl Dixon Winterhouse Consulting Ltd http://www.winterhouseconsulting.com ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZeoServer, multiple storages and open file handles
Hi Jim Do you have any idea as to why people recommend against many storages under a single Zeo? Also can increasing the invalidation-queue-size help this if there is memory to spare on the machine? Tim On 23 October 2012 00:53, Jim Fulton j...@zope.com wrote: On Sun, Oct 21, 2012 at 8:48 PM, Tim Godfrey t...@obsidian.com.au wrote: Hi Jon Thanks for your response. Is that something that has been done in a later version of Zeoserver than mine (ZODB3-3.10.3)? No. Every time I've tried to switch to poll in a asyncore, I've had problems. It this your recommended action for the issue I'm having or are there still some configuration changes I can make? Many people have recommended not hosting many storages in a single process. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm -- Tim Godfrey Obsidian Consulting Group P: +61 3 9355 7844 F: +61 3 9350 4097 E: t...@obsidian.com.au W: http://www.obsidian.com.au/ ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZeoServer, multiple storages and open file handles
On Sun, Oct 21, 2012 at 8:48 PM, Tim Godfrey t...@obsidian.com.au wrote: Hi Jon Thanks for your response. Is that something that has been done in a later version of Zeoserver than mine (ZODB3-3.10.3)? No. Every time I've tried to switch to poll in a asyncore, I've had problems. It this your recommended action for the issue I'm having or are there still some configuration changes I can make? Many people have recommended not hosting many storages in a single process. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZeoServer, multiple storages and open file handles
Hi Jon Thanks for your response. Is that something that has been done in a later version of Zeoserver than mine (ZODB3-3.10.3)? It this your recommended action for the issue I'm having or are there still some configuration changes I can make? Tim On 20 October 2012 19:52, Jon Dyte j...@totient.co.uk wrote: On 18/10/12 03:29, Tim Godfrey wrote: Hi all We have a customer on Plone 4.1 and recently we started to see a great increase in the number of open file descriptors the Zeoserver process uses. We have a pretty big system distributed over many servers as the Plone system gets a lot of requests. Currently we are running a Zeoserver with 30 storages and have 16 Zope clients that connect to it. I've had Zeoserver in debug mode to try and get some more insight into the problem and I'm seeing a lot of these errors: Exception in thread Thread-575: Traceback (most recent call last): File /opt/informa/Python-2.6/lib/**python2.6/threading.py, line 532, in __bootstrap_inner self.run() File /opt/informa/Python-2.6/lib/**python2.6/threading.py, line 484, in run self.__target(*self.__args, **self.__kwargs) File /opt/informa/buildout-cache/**eggs/ZODB3-3.10.3-py2.6-linux-** x86_64.egg/ZEO/zrpc/**connection.py, line 661, in server_loop asyncore.poll(30.0, map) File /opt/informa/Python-2.6/lib/**python2.6/asyncore.py, line 140, in poll r, w, e = select.select(r, w, e, timeout) ValueError: filedescriptor out of range in select() if the server_loop function in connection.py was changed to call asyncore.poll2 as far as I can tell the descriptor out of range should go away as it uses the poll call rather than select. Jon __**_ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/**listinfo/zodb-devhttps://mail.zope.org/mailman/listinfo/zodb-dev -- Tim Godfrey Obsidian Consulting Group P: +61 3 9355 7844 F: +61 3 9350 4097 E: t...@obsidian.com.au W: http://www.obsidian.com.au/ ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZeoServer, multiple storages and open file handles
On 18/10/12 03:29, Tim Godfrey wrote: Hi all We have a customer on Plone 4.1 and recently we started to see a great increase in the number of open file descriptors the Zeoserver process uses. We have a pretty big system distributed over many servers as the Plone system gets a lot of requests. Currently we are running a Zeoserver with 30 storages and have 16 Zope clients that connect to it. I've had Zeoserver in debug mode to try and get some more insight into the problem and I'm seeing a lot of these errors: Exception in thread Thread-575: Traceback (most recent call last): File /opt/informa/Python-2.6/lib/python2.6/threading.py, line 532, in __bootstrap_inner self.run() File /opt/informa/Python-2.6/lib/python2.6/threading.py, line 484, in run self.__target(*self.__args, **self.__kwargs) File /opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/zrpc/connection.py, line 661, in server_loop asyncore.poll(30.0, map) File /opt/informa/Python-2.6/lib/python2.6/asyncore.py, line 140, in poll r, w, e = select.select(r, w, e, timeout) ValueError: filedescriptor out of range in select() if the server_loop function in connection.py was changed to call asyncore.poll2 as far as I can tell the descriptor out of range should go away as it uses the poll call rather than select. Jon ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZeoServer, multiple storages and open file handles
Having a single ZEO server handling more than one or two storages is
not usually a good idea. ZEO does not handle each storage in a
separate thread, so you're underusing multiple CPUs if you have them.
Multiple ZEO servers would also spread the open sockets among them.
There could still be a leak, but it'd take longer to exhaust file
descriptors.
On Wed, Oct 17, 2012 at 11:29 PM, Tim Godfrey t...@obsidian.com.au wrote:
Hi all
We have a customer on Plone 4.1 and recently we started to see a great
increase in the number of open file descriptors the Zeoserver process uses.
We have a pretty big system distributed over many servers as the Plone
system gets a lot of requests. Currently we are running a Zeoserver with 30
storages and have 16 Zope clients that connect to it.
I've had Zeoserver in debug mode to try and get some more insight into the
problem and I'm seeing a lot of these errors:
Exception in thread Thread-575:
Traceback (most recent call last):
File /opt/informa/Python-2.6/lib/python2.6/threading.py, line 532, in
__bootstrap_inner
self.run()
File /opt/informa/Python-2.6/lib/python2.6/threading.py, line 484, in
run
self.__target(*self.__args, **self.__kwargs)
File
/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/zrpc/connection.py,
line 661, in server_loop
asyncore.poll(30.0, map)
File /opt/informa/Python-2.6/lib/python2.6/asyncore.py, line 140, in
poll
r, w, e = select.select(r, w, e, timeout)
ValueError: filedescriptor out of range in select()
Exception in thread Thread-576:
Traceback (most recent call last):
File /opt/informa/Python-2.6/lib/python2.6/threading.py, line 532, in
__bootstrap_inner
self.run()
File /opt/informa/Python-2.6/lib/python2.6/threading.py, line 484, in
run
self.__target(*self.__args, **self.__kwargs)
File
/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/zrpc/connection.py,
line 661, in server_loop
asyncore.poll(30.0, map)
File /opt/informa/Python-2.6/lib/python2.6/asyncore.py, line 140, in
poll
r, w, e = select.select(r, w, e, timeout)
ValueError: filedescriptor out of range in select()
There's also a lot of these lines in zeoserver.log:
2012-10-18T12:42:08 new connection ('10.221.0.144', 34919):
ManagedServerConnection ('10.221.0.144', 34919)
2012-10-18T12:42:08 new connection ('10.221.0.64', 41539):
ManagedServerConnection ('10.221.0.64', 41539)
Each time a new connection is logged further open file descriptors are used.
There are also these errors in the Zope clients:
2012-10-18T13:20:01 ERROR ZEO.zrpc (7746) CW: error in testConnection
(('10.221.0.247', 8101))
Traceback (most recent call last):
File
/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/zrpc/client.py,
line 595, in test_connection
self.preferred = self.client.testConnection(self.conn)
File
/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/ClientStorage.py,
line 565, in testConnection
stub = self.StorageServerStubClass(conn)
File
/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/ServerStub.py,
line 386, in stub
raise ValueError(Timeout waiting for protocol handshake)
ValueError: Timeout waiting for protocol handshake
At time of writing the Zeoserver process is using 1604 file descriptors but
that number will continue to grow. I suppose I'm wondering why it's always
creating new fds and not closing old ones. Also whether there are changes I
can make to my configuration to stop these errors and growing number of file
descriptors.
If Zeoserver is allowed to keep running it will eventually hit the operating
system limit of open fds and start raising Too Many Open File Exceptions
which causes havoc with transactions are I'm sure you would know.
Is anyone able to offer some advice? I'm more than happy to provide more
information if required.
Thanks,
Tim
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZeoServer, multiple storages and open file handles
On Thu, Oct 18, 2012 at 3:09 PM, Leonardo Rochael Almeida leoroch...@gmail.com wrote: Thanks for pitching in with an answer! Having a single ZEO server handling more than one or two storages is not usually a good idea. ZEO does not handle each storage in a separate thread, so you're underusing multiple CPUs if you have them. Nit pick: ZEO handles each client connection and storage in a separate thread. (So 30 storages and 16 clients means 480 threads :) It is Python's GIL that prevents effective use of multiple processors. ZEO goes out if it's way to let I/O C code run concurrently (since I/O isn't subject to the GIL) and I've seen ZEO storage servers use up to 200% CPU on 4-core boxes (2 cores worth, IOW). Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm ___ For more information about ZODB, see http://zodb.org/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZeoServer, multiple storages and open file handles
Hi
Thanks for your responses thus far.
For the most part the zeoserver is performing quite well. Very low memory
and CPU usage.
I really think my problem lies in the fact that the Zope clients are
reporting these sort of messages quite regularly when communicating to ZEO:
2012-10-19T10:15:02 INFO ZEO.ClientStorage storage_d06_zeostorage Testing
connection ManagedClientConnection ('10.221.0.247', 8101)
--
2012-10-19T10:15:12 ERROR ZEO.zrpc (8356) CW: error in testConnection
(('10.221.0.247', 8101))
Traceback (most recent call last):
File
/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/zrpc/client.py,
line 595, in test_connection
self.preferred = self.client.testConnection(self.conn)
File
/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/ClientStorage.py,
line 565, in testConnection
stub = self.StorageServerStubClass(conn)
File
/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/ServerStub.py,
line 386, in stub
raise ValueError(Timeout waiting for protocol handshake)
ValueError: Timeout waiting for protocol handshake
Each time that happens, more open file descriptors are used by zeo and as
far as I'm aware are never released unless the Zope client shuts down.
I have upped zeo's invalidation-queue-size to 4000 (not sure what my upper
limit is there) to see if that reduces the growth of open file handles. So
far that seems to be the case. 16 Zope clients connected with only 413 file
descriptors in use.
Not sure if any of these observations are related however?
Tim
On 19 October 2012 06:29, Jim Fulton j...@zope.com wrote:
On Thu, Oct 18, 2012 at 3:09 PM, Leonardo Rochael Almeida
leoroch...@gmail.com wrote:
Thanks for pitching in with an answer!
Having a single ZEO server handling more than one or two storages is
not usually a good idea. ZEO does not handle each storage in a
separate thread, so you're underusing multiple CPUs if you have them.
Nit pick: ZEO handles each client connection and storage in a separate
thread.
(So 30 storages and 16 clients means 480 threads :)
It is Python's GIL that prevents effective use of multiple processors.
ZEO goes out if it's way to let I/O C code run concurrently (since I/O
isn't
subject to the GIL) and I've seen ZEO storage servers use up to 200% CPU
on 4-core boxes (2 cores worth, IOW).
Jim
--
Jim Fulton
http://www.linkedin.com/in/jimfulton
Jerky is better than bacon! http://zo.pe/Kqm
--
Tim Godfrey
Obsidian Consulting Group
P: +61 3 9355 7844
F: +61 3 9350 4097
E: t...@obsidian.com.au
W: http://www.obsidian.com.au/
___
For more information about ZODB, see http://zodb.org/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
