Re: Py2.7/FreeBSD: maximum number of open files
In article Christian Heimes wrote: >Am 14.11.2011 19:28, schrieb Tobias Oberstein: >> Thanks! This is probably the most practical option I can go. >> >> I've just tested: the backported new IO on Python 2.7 will indeed >> open >32k files on FreeBSD. It also creates the files much faster. >> The old, non-monkey-patched version was getting slower and >> slower as more files were opened/created .. > >I wonder what's causing the O(n^2) behavior. Is it the old file type or >BSD's fopen() fault? It is code in libc. My old stdio (still in use on FreeBSD) was never designed to be used in situations with more than roughly 256 file descriptors -- hence the "short" in the file number field. (The OS used to be full of other places that kept the maximum allowable file descriptor fairly small, such as the on-stack copies of fd_set objects in select() calls.) You will want to redesign the code that finds or creates a free "FILE" object, and probably some of the things that work with line-buffered FILEs (specifically calls to _fwalk() when reading from a line-buffered stream). -- In-Real-Life: Chris Torek, Wind River Systems Intel require I note that my opinions are not those of WRS or Intel Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W) +1 801 277 2603 email: gmail (figure it out) http://web.torek.net/torek/index.html -- http://mail.python.org/mailman/listinfo/python-list
Re: Py2.7/FreeBSD: maximum number of open files
Am 14.11.2011 19:28, schrieb Tobias Oberstein: > Thanks! This is probably the most practical option I can go. > > I've just tested: the backported new IO on Python 2.7 will indeed > open >32k files on FreeBSD. It also creates the files much faster. > The old, non-monkey-patched version was getting slower and > slower as more files were opened/created .. I wonder what's causing the O(n^2) behavior. Is it the old file type or BSD's fopen() fault? > There seem to be slight differences though: > > Non-monkey patched: I can write to the file a non-Unicode string, > even when the file was opened non-Binary. > > With monkey patch: either open the file Binary-mode, or > write Unicode strings .. Python 3.x doesn't collate data and text but distinguishes between bytes (str type in Python 2.x) and unicode. You can't write unicode to a binary file and str to a text file. See it as an opportunity! You are taking the first step towards Python 3 compatibility. :) Christian -- http://mail.python.org/mailman/listinfo/python-list
AW: Py2.7/FreeBSD: maximum number of open files
> > I just confirmed that the bug is even there for FreeBSD 9 RC1 !
> >
> > This is most unfortunate. Seriously.
>
> W00t, that sucks! You could migrate to another BSD (NetBSD) or Linux ... :)
No, thanks;)
> > I am running out of options, since I am willing to make my stuff
> > Python 3 compatible, but Twisted is not yet there.
> >
> > Using the backported new IO on Python 2.7 will not make open()
> automatically use the new IO, will it?
>
> No, the open() function of Python 2.7 will still use the file class which in
> return uses fopen(). You could try to monkey patch the built-in
> open() function. It's mostly API compatible with the current open()
> function:
>
> >>> import io, __builtin__
> >>> __builtin__.open = io.open
>
> It works as long as no codes checks for isinstance(obj, file). If your app
> only
> has to worry about log files, you might want to overwrite the
> _open() method of logging.FileHandler and its subclasses.
>
Thanks! This is probably the most practical option I can go.
I've just tested: the backported new IO on Python 2.7 will indeed
open >32k files on FreeBSD. It also creates the files much faster.
The old, non-monkey-patched version was getting slower and
slower as more files were opened/created ..
There seem to be slight differences though:
Non-monkey patched: I can write to the file a non-Unicode string,
even when the file was opened non-Binary.
With monkey patch: either open the file Binary-mode, or
write Unicode strings ..
I need to see if / what breaks in Twisted.
I can handle my own code .. no problem.
Thanks alot!!
import io, __builtin__
__builtin__.open = io.open
import resource
max = resource.getrlimit(resource.RLIMIT_NOFILE)
cnt = 0
print "maximum FDs", max
max = 33000
fds = []
while cnt < max:
f = open("/tmp/test1/test_%d" % cnt, "wb")
f.write("test")
fds.append(f)
cnt += 1
if cnt % 1000 == 0:
print "opened %d files" % cnt
print "ok, created %d files" % cnt
--
http://mail.python.org/mailman/listinfo/python-list
Re: Py2.7/FreeBSD: maximum number of open files
Am 14.11.2011 18:46, schrieb Tobias Oberstein: > I just confirmed that the bug is even there for FreeBSD 9 RC1 ! > > This is most unfortunate. Seriously. W00t, that sucks! You could migrate to another BSD (NetBSD) or Linux ... :) > I am running out of options, since I am willing to make my stuff Python 3 > compatible, > but Twisted is not yet there. > > Using the backported new IO on Python 2.7 will not make open() automatically > use the new IO, will it? No, the open() function of Python 2.7 will still use the file class which in return uses fopen(). You could try to monkey patch the built-in open() function. It's mostly API compatible with the current open() function: >>> import io, __builtin__ >>> __builtin__.open = io.open It works as long as no codes checks for isinstance(obj, file). If your app only has to worry about log files, you might want to overwrite the _open() method of logging.FileHandler and its subclasses. Christian -- http://mail.python.org/mailman/listinfo/python-list
AW: Py2.7/FreeBSD: maximum number of open files
> > > > I need 50k sockets + 100 files. > > > > > > Thus, this is even more strange: the Python (a Twisted service) > > > > will happily accept 50k sockets, but as soon as you do open() a file, > > > > it'll > bail out. > > > > > A limit of 32k smells like a overflow in a signed int. Perhaps your > > > system is able and configured to handle more than 32k FDs but you > > > hit an artificial limit because some C code or API has a overflow. > > > This seems to be a known bug in > > > FreeBSDhttp://lists.freebsd.org/pipermail/freebsd-bugs/2010- > > > July/040689.html > > > > This is unbelievable. > > > > I've just tested: the bug (in libc) is still there on FreeBSD 8.2 p3 > > ... both on i386 _and_ amd64. > > > > Now I'm f***d;( > > > > A last chance: is it possible to compile Python for not using libc > > fopen(), but the Posix open()? > > > > Thanks anyway for this hint! > > Have you tried/or is it possible to get your 100 or whatever files first, > before > your sockets? If I only needed to open a fixed set of files, that might be a workaround. However, this is not the case. I.e. Twisted will do log switching and create/open a new file when the 50k sockets are already there. I just confirmed that the bug is even there for FreeBSD 9 RC1 ! This is most unfortunate. Seriously. I am running out of options, since I am willing to make my stuff Python 3 compatible, but Twisted is not yet there. Using the backported new IO on Python 2.7 will not make open() automatically use the new IO, will it? -- http://mail.python.org/mailman/listinfo/python-list
Re: Py2.7/FreeBSD: maximum number of open files
On Nov 14, 5:03 pm, Tobias Oberstein wrote: > > > I need 50k sockets + 100 files. > > > > Thus, this is even more strange: the Python (a Twisted service) will > > > happily accept 50k sockets, but as soon as you do open() a file, it'll > > > bail out. > > > A limit of 32k smells like a overflow in a signed int. Perhaps your system > > is > > able and configured to handle more than 32k FDs but you hit an artificial > > limit > > because some C code or API has a overflow. This seems to be a known bug in > > FreeBSDhttp://lists.freebsd.org/pipermail/freebsd-bugs/2010- > > July/040689.html > > This is unbelievable. > > I've just tested: the bug (in libc) is still there on FreeBSD 8.2 p3 ... both > on i386 > _and_ amd64. > > Now I'm f***d;( > > A last chance: is it possible to compile Python for not using libc fopen(), > but the Posix open()? > > Thanks anyway for this hint! Have you tried/or is it possible to get your 100 or whatever files first, before your sockets? hth Jon -- http://mail.python.org/mailman/listinfo/python-list
Re: Py2.7/FreeBSD: maximum number of open files
Am 14.11.2011 18:03, schrieb Tobias Oberstein: > This is unbelievable. > > I've just tested: the bug (in libc) is still there on FreeBSD 8.2 p3 ... both > on i386 > _and_ amd64. > > Now I'm f***d;( > > A last chance: is it possible to compile Python for not using libc fopen(), > but the Posix open()? > > Thanks anyway for this hint! A fix would break the ABI compatibility. I guess that you won't see a fix for the issue until FreeBSD 9.0 is released. And no, you can't re-compile Python to use the open() API instead of fopen(). The built-in file type is developed around the file pointer API, not the file descriptor API. Luckily Python 2.7 has a backport of Python 3.x new IO library. The new IO API doesn't use file pointers at all and implements its own layer around file descriptors. Good luck! Christian -- http://mail.python.org/mailman/listinfo/python-list
AW: Py2.7/FreeBSD: maximum number of open files
> > I need 50k sockets + 100 files. > > > > Thus, this is even more strange: the Python (a Twisted service) will > > happily accept 50k sockets, but as soon as you do open() a file, it'll bail > > out. > > A limit of 32k smells like a overflow in a signed int. Perhaps your system is > able and configured to handle more than 32k FDs but you hit an artificial > limit > because some C code or API has a overflow. This seems to be a known bug in > FreeBSD http://lists.freebsd.org/pipermail/freebsd-bugs/2010- > July/040689.html This is unbelievable. I've just tested: the bug (in libc) is still there on FreeBSD 8.2 p3 ... both on i386 _and_ amd64. Now I'm f***d;( A last chance: is it possible to compile Python for not using libc fopen(), but the Posix open()? Thanks anyway for this hint! -- http://mail.python.org/mailman/listinfo/python-list
Re: Py2.7/FreeBSD: maximum number of open files
Am 14.11.2011 17:36, schrieb Tobias Oberstein: > This is a dedicated machine doing nothing else .. I'm monitoring global FD > usage > > sysctl kern.openfiles > > and it's way beyond the configured limit > > $ ulimit -n > 20 Apparently you did everything right here. Well, it was worth the try. ;) > It's not for files: > > This is a network service .. I tested it with up to 50k TCP connections .. > however > at this point, when the service tries to open a file, it'll bail out. > > Sockets+Files both contribute to open FDs. > > I need 50k sockets + 100 files. > > Thus, this is even more strange: the Python (a Twisted service) will happily > accept 50k sockets, but as soon as you do open() a file, it'll bail out. A limit of 32k smells like a overflow in a signed int. Perhaps your system is able and configured to handle more than 32k FDs but you hit an artificial limit because some C code or API has a overflow. This seems to be a known bug in FreeBSD http://lists.freebsd.org/pipermail/freebsd-bugs/2010-July/040689.html Christian -- http://mail.python.org/mailman/listinfo/python-list
AW: Py2.7/FreeBSD: maximum number of open files
> I'm not familiar with BSD but Linux has similar Kernel options. The kernel > options might be *global* flags to set the total upper limit of open file > descriptors for the entire system, not for a single process. > Also on Linux "ulimit" doesn't display the fd limit. You have to use "ulimit > -n". This is a dedicated machine doing nothing else .. I'm monitoring global FD usage sysctl kern.openfiles and it's way beyond the configured limit $ ulimit -n 20 > > Why do you need more than 32k file descriptors anyway? It's an insanely high It's not for files: This is a network service .. I tested it with up to 50k TCP connections .. however at this point, when the service tries to open a file, it'll bail out. Sockets+Files both contribute to open FDs. I need 50k sockets + 100 files. Thus, this is even more strange: the Python (a Twisted service) will happily accept 50k sockets, but as soon as you do open() a file, it'll bail out. -- http://mail.python.org/mailman/listinfo/python-list
Re: Py2.7/FreeBSD: maximum number of open files
Am 14.11.2011 16:57, schrieb Tobias Oberstein: > I am trying to convince Python to open more than 32k files .. this is on > FreeBSD. > > Now I know I have to set appropriate limits .. I did: > > $ sysctl kern.maxfiles > kern.maxfiles: 204800 > $ sysctl kern.maxfilesperproc > kern.maxfilesperproc: 20 > $ sysctl kern.maxvnodes > kern.maxvnodes: 20 > $ ulimit > unlimited > > Here is what happens with a Python freshly built from sources .. it'll tell > me I can open 200k files .. but will bail out at 32k: I'm not familiar with BSD but Linux has similar Kernel options. The kernel options might be *global* flags to set the total upper limit of open file descriptors for the entire system, not for a single process. Also on Linux "ulimit" doesn't display the fd limit. You have to use "ulimit -n". Why do you need more than 32k file descriptors anyway? It's an insanely high amount of FDs. Most programs need less than 100 and the default value of 1024 on my Linux servers is usually high enough. I've never increased the fd limit over 8192 and our biggest installation servers more than 80 TB data in about 20 to 25 million files. Christian -- http://mail.python.org/mailman/listinfo/python-list
Py2.7/FreeBSD: maximum number of open files
I am trying to convince Python to open more than 32k files .. this is on
FreeBSD.
Now I know I have to set appropriate limits .. I did:
$ sysctl kern.maxfiles
kern.maxfiles: 204800
$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 20
$ sysctl kern.maxvnodes
kern.maxvnodes: 20
$ ulimit
unlimited
Here is what happens with a Python freshly built from sources .. it'll tell me
I can open 200k files .. but will bail out at 32k:
$ ./local/bin/python -V
Python 2.7.2
$ ./local/bin/python
Python 2.7.2 (default, Nov 14 2011, 16:41:56)
[GCC 4.2.1 20070719 [FreeBSD]] on freebsd8
Type "help", "copyright", "credits" or "license" for more information.
>>> import resource
>>> resource.getrlimit(resource.RLIMIT_NOFILE)
(20L, 20L)
>>> resource.getrlimit(resource.RLIMIT_OFILE)
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'module' object has no attribute 'RLIMIT_OFILE'
>>> import resource
>>>
>>> max = resource.getrlimit(resource.RLIMIT_NOFILE)
>>> cnt = 0
>>> print "maximum FDs", max
maximum FDs (20L, 20L)
fds = []
while cnt < max:
f = open("/tmp/test1/test_%d" % cnt, "w")
f.write("test")
fds.append(f)
cnt += 1
if cnt % 1000 == 0:
print "opened %d files" % cnt
print "ok, created %d files" % cnt
>>>
>>> fds = []
>>>
>>> while cnt < max:
...f = open("/tmp/test1/test_%d" % cnt, "w")
...f.write("test")
...fds.append(f)
...cnt += 1
...if cnt % 1000 == 0:
... print "opened %d files" % cnt
...
opened 1000 files
opened 2000 files
opened 3000 files
opened 4000 files
opened 5000 files
opened 6000 files
opened 7000 files
opened 8000 files
opened 9000 files
opened 1 files
opened 11000 files
opened 12000 files
opened 13000 files
opened 14000 files
opened 15000 files
opened 16000 files
opened 17000 files
opened 18000 files
opened 19000 files
opened 2 files
opened 21000 files
opened 22000 files
opened 23000 files
opened 24000 files
opened 25000 files
opened 26000 files
opened 27000 files
opened 28000 files
opened 29000 files
opened 3 files
opened 31000 files
opened 32000 files
Traceback (most recent call last):
File "", line 2, in
IOError: [Errno 24] Too many open files: '/tmp/test1/test_32765'
>>> print "ok, created %d files" % cnt
ok, created 32765 files
>>>
--
http://mail.python.org/mailman/listinfo/python-list
