Thank you, Tres! I think this will do even better for the short term. I am
now trying to start up everything with
stdout_logfile=NONE and stderr_logfile=NONE in each of my [program:]
stanzas.
I can do this because all of my monitored processes make their own logs (log4j
with a FileAppender and a ConsoleAppender), and these files have essentially
the same content at the stdout/stderr files that supervisor writes. The only
thing I lose is the ability to use the 'tail -f' links from the webpage ("No
file for :jobname"), which I honestly don't think any of my users are using.
This lowers my fd consumption (3 fds per process instead of 5). With this
savings, I should be able to scale up to ~335-340 processes and still live
within the 1024 fd limit.
tlj
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Tres Seaver
Sent: Thursday, September 08, 2011 1:49 PM
To: [email protected]
Subject: Re: [Supervisor-users] Supervisord needs more than 1024 file
descriptors?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 09/08/2011 01:12 PM, Timothy Jones wrote:
> We're using supervisord to monitor 197 processes. We just added 10
> more, for a total of 207, and started Server Error 500 when trying
> to view the status webpage. I traced the error 500 to python getting
> an "out of file descriptors" error (which I know from experience is
> normally only 1024). This seems to be legitimate, because I see 5
> fds being used for each process, so not counting basic overhead and
> the rpc/http listeners, I needed 985 (197*5), and now I need 1035
> (207*5).
>
> So I raised the soft limit on file descriptors to 1536 (a 50%
> increase) and raised the hard limit to 65,000, by creating the
> following file at /etc/security/limits.d/60-nofile-limit.conf,
> containing these lines:
>
> fs.file-max = 65000 * soft nofile 1536 * hard nofile 65000
>
> I logged out, logged back in, verified that 'ulimit -n' does indeed
> show 1536 instead of 1024. I restarted supervisor again, and I
> added the argument: "--minfds=1536".
>
> I told supervisor to start all the processes. When it got up to
> 1024 descriptors, I expected it to continue. Instead I get a
> different error, this time from Python itself, resulting in an exit:
> ValueError: filedescriptor out of range in select()
>
> Is python's select() call simply saying this because some of the fd
> values are higher than 1024? Is there some other python-specific
> thing I need to raise so it will accept fd values higher than 1024?
> Supervisord seems to know something about it, because I got the
> message "Increased RLIMIT_NOFILE limit to 1536" in the
> supervisor.log.
Python's 'select' module wraps the base C library version, which depends
on a compiled-in hard limit, FD_SETSIZE. That limit defaults to 1024 on
POSIX systems, and 64 on Windows: Python bumps the value to 512 in that
case. If you are running an OS whose libc allows applications to define
a different FD_SETSIZE, you could try rebuilding Python with e.g.:
-DFD_SETSIZE=2048
You could also try reducing the number of open FDs per process, e.g. by
having the managed processes log directly to files, and configuring
supervisor not to manage their stdout / stderr.
Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 [email protected]
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk5o//UACgkQ+gerLs4ltQ7PNwCfQw0DrGNPNay88Vdvm39FmMcN
/xkAoMj4go0HQz2ATCex05yDIGiwWkWz
=l83m
-----END PGP SIGNATURE-----
_______________________________________________
Supervisor-users mailing list
[email protected]
http://lists.supervisord.org/mailman/listinfo/supervisor-users
_______________________________________________
Supervisor-users mailing list
[email protected]
http://lists.supervisord.org/mailman/listinfo/supervisor-users
