2009/3/6 Brian Sutherland <[email protected]>:
>
> On Mar 6, 11:53 am, Graham Dumpleton <[email protected]>
> wrote:
>> 2009/3/6 Brian Sutherland <[email protected]>:
>>
>> > I've managed to make a minimal wsgi application that shows off a
>> > problem we encountered in production. It looks like mod_wsgi is
>> > leaking the temporary files we give to it. Perhaps we are doing
>> > something wrong, but I couldn't find any docs against passing
>> > tempfile.mkstemp files to mod_wsgi. Curiously, creating the file with
>> > tempfile.TemporaryFile rather than tempfile.mkstemp has allowed us to
>> > work around the issue.
>>
>> > Basically running ab against this application:
>>
>> > #!/usr/bin/python
>> > import tempfile
>> > import os
>>
>> > def application(environ, start_response):
>> > status = '200 OK'
>> > output = 'Hello World!'
>>
>> > response_headers = [('Content-type', 'text/plain'),
>> > ('Content-Length', str(len(output)))]
>>
>> > start_response(status, response_headers)
>>
>> > # Make a temporary file and write data to it
>> > _, filename = tempfile.mkstemp()
>>
>> This returns a file descriptor which you are assigning to '_' variable
>> but you aren't closing it. This means that when it is closed is at the
>> mercy of the Python garbage collector. They thus aren't being closed
>> promptly and you are running out of file descriptors.
>
> The problem is that they are never closed, after hours, even while
> other requests succeed. To try discount your idea I changed my test
> script to this, and got the same result, after ~1000 requests it
> starts failing:
>
> #!/usr/bin/python
> import tempfile
> import os
> import gc
>
> def application(environ, start_response):
> status = '200 OK'
> output = 'Hello World!'
>
> response_headers = [('Content-type', 'text/plain'),
> ('Content-Length', str(len(output)))]
>
> start_response(status, response_headers)
>
> # Make a temporary file and write data to it
> filedesc, filename = tempfile.mkstemp()
> del filedesc
> gc.collect()
The use:
os.close(filedesc)
if it isn't a file object.
It being a normal file descriptor and not file object object makes it
more obvious that it would be a problem as it would never be closed as
side affect of any garbage collection.
>
> f = open(filename, 'wb')
> f.write(output)
> f.close()
>
> # Make a deleted read-only file to pass to mod_wsgi
> readf = open(filename, 'rb')
> os.remove(filename)
>
> wrapper = environ.get('wsgi.file_wrapper')
> return wrapper(readf)
>
>
>> > f = open(filename, 'wb')
>> > f.write(output)
>> > f.close()
>>
>> > # Make a deleted read-only file to pass to mod_wsgi
>> > readf = open(filename, 'rb')
>> > os.remove(filename)
>>
>> You probably want to use something like:
>>
>> f, filename = tempfile.mkstemp()
>> f.write(output)
>> f.flush()
>> f.seek(0)
>>
>> Then use 'f' as argument to file wrapper.
>
> Doesn't seem to work, f is an "OS-level handle to an open file (as
> would be returned by os.open())":
Then try:
f = os.fdopen(f)
Which is what TemporaryFile does.
Graham
>>>> import tempfile
>>>> f,filename = tempfile.mkstemp()
>>>> f.write('bobo')
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> AttributeError: 'int' object has no attribute 'write'
>
>
>>
>> So, my questions about MPM and mode aren't relevant.
>>
>> Graham
>>
>> > wrapper = environ.get('wsgi.file_wrapper')
>> > return wrapper(readf)
>>
>> > Causes these errors:
>>
>> > [Fri Mar 06 09:50:16 2009] [error] [client 62.57.1.12] mod_wsgi
>> > (pid=18665): Exception occurred processing WSGI script '/etc/apache2/
>> > wsgi/wsgitest.wsgi'.
>> > [Fri Mar 06 09:50:16 2009] [error] Traceback (most recent call last):
>> > [Fri Mar 06 09:50:16 2009] [error] File "/etc/apache2/wsgi/
>> > wsgitest.wsgi", line 15, in application
>> > [Fri Mar 06 09:50:16 2009] [error] File "tempfile.py", line 302, in
>> > mkstemp
>> > [Fri Mar 06 09:50:16 2009] [error] File "tempfile.py", line 236, in
>> > _mkstemp_inner
>> > [Fri Mar 06 09:50:16 2009] [error] OSError: [Errno 24] Too many open
>> > files: '/tmp/tmp3zONgs'
>>
>> > Because you very quickly get to the per-process file limit:
>>
>> > load:/etc/apache2/wsgi# ls /proc/18665/fd | wc -l
>> > 1023
>>
>> > We're running python 2.4.4, apache 2.2.3-4+etch6 on Debian Etch and
>> > have seen this with mod_wsgi 2.1 and 2.3.
>>
>> > Many Thanks,
>> > Brian
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/modwsgi?hl=en
-~----------~----~----~----~------~----~------~--~---
