Re: [ZODB-Dev] ZEO server crash
Hello. Sorry for the lack of information I provided. I am running a ZEO server used by a CherryPy application - which btw. can be seen here: http://orango.amix.dk/ Everything runs fine... For about one hours or two. Then ZEO crashes with that error. I restart ZEO and restart the webapplication. They again run and BAM, after one hour or two ZEO crashes with the exact same error. Right now, when I count the number of files those two have open (ZEO=22437 and CherryPy application=22465): lsof | grep 22437 | wc -l 872 lsof | grep 22465 | wc -l 874 Which is kind of very high... and they are very equal. WTF!?... I have just found out that the web applications opens some connections, which it does not close. Problem is now solved.. Sorry :( And thanks for your response! ZEO and ZODB kicks ass btw..! I am truly amazed :) -- kind regards: 4mi(r|x) [http://amix.dk] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] ZEO server crash
FWIW, the shell also limits the number of open files that each user can have open. The default is typically a puny 1024 files (you can check this by running ulimit -n as the user under which you're running the ZEO server). To bump this up, under Linux you typically need to set a line like this in your /etc/security/limits.conf file and restart the system: * soft nofile 65535 * hard nofile 65535 Obviously if one of your apps (ZEO or another app that happens to run as this user under the same parent shell) is leaking file descriptors, this is only a stopgap. You can probably do "watch cat /proc/sys/fs/file-nr" to see how the number of filedescriptors grows; maybe you can get a sense of what's causing it by interactive inspection. - C On Thu, 2005-03-31 at 18:34, Tim Peters wrote: > [4mi(r|x)] > > I am using ZEO as a server on a Debian linux 2.6 machine. The ZEO version > > is 3.3. > > > > I get following error: > > error: uncaptured python exception, closing channel > >0xb79ec56c> (exceptions.OSError:[Errno 24] Too many open files > > > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ThreadedAsync/LoopCal > lback.py|poll|129] > > [/usr/lib/python2.3/asyncore.py|handle_read_event|384] > > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/server.py|ha > ndle_accept|58] > > > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/StorageServer.py| > new_connection|803] > > > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/connection.p > y|__init__|566] > > > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/connection.p > y|__init__|159] > > > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/connection.p > y|_prepare_async|421] > > > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ThreadedAsync/LoopCal > lback.py|register_loop_callback|64] > > > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/connection.p > y|set_async|426] > > > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/trigger.py|_ > _init__|55]) > > > > I have tried to set the /proc/sys/fs/files-max up but that did not help > > :-( > > Meaning what? You get exactly the same traceback every time? Every time > you try to connect to ZEO? Once an hour? Once a day? Too much telepathy > required here. Are you using Zope? Is this a standalone application? Are > other apps on the box mucking with a large number of files? Etc. > > > Is there any fix for this? > > I don't know. The only other report I found was from 2001, and didn't > appear to be resolved at the time: > > http://mail.zope.org/pipermail/zodb-dev/2001-October/001616.html > > Since others don't report "Too many open files" errors, presumption has to > favor that you're doing something unusual, but I can't guess what. Line 55 > in trigger.py is > > r, w = self._fds = os.pipe() > > so all we can tell from the traceback is that the OS ran out of file > descriptors and so can't create a new pipe. The descriptors opened there > are closed again by the trigger.close() method, so I suppose that if the > latter isn't getting called, open descriptors would build up. > > ___ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zodb-dev > ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] ZEO server crash
[4mi(r|x)] > I am using ZEO as a server on a Debian linux 2.6 machine. The ZEO version > is 3.3. > > I get following error: > error: uncaptured python exception, closing channel > (exceptions.OSError:[Errno 24] Too many open files > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ThreadedAsync/LoopCal lback.py|poll|129] > [/usr/lib/python2.3/asyncore.py|handle_read_event|384] > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/server.py|ha ndle_accept|58] > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/StorageServer.py| new_connection|803] > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/connection.p y|__init__|566] > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/connection.p y|__init__|159] > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/connection.p y|_prepare_async|421] > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ThreadedAsync/LoopCal lback.py|register_loop_callback|64] > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/connection.p y|set_async|426] > [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/trigger.py|_ _init__|55]) > > I have tried to set the /proc/sys/fs/files-max up but that did not help > :-( Meaning what? You get exactly the same traceback every time? Every time you try to connect to ZEO? Once an hour? Once a day? Too much telepathy required here. Are you using Zope? Is this a standalone application? Are other apps on the box mucking with a large number of files? Etc. > Is there any fix for this? I don't know. The only other report I found was from 2001, and didn't appear to be resolved at the time: http://mail.zope.org/pipermail/zodb-dev/2001-October/001616.html Since others don't report "Too many open files" errors, presumption has to favor that you're doing something unusual, but I can't guess what. Line 55 in trigger.py is r, w = self._fds = os.pipe() so all we can tell from the traceback is that the OS ran out of file descriptors and so can't create a new pipe. The descriptors opened there are closed again by the trigger.close() method, so I suppose that if the latter isn't getting called, open descriptors would build up. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
