How is the file being accessed from the NFS file system being delivered back to the client?
If the code is ultimately using wsgi.file_wrapper, try setting: WSGIEnableSendfile Off The EnableSendfile directive only applies to static files Apache itself serves. Those delivered by mod_wsgi using wsgi.file_wrapper are configured by separate WSGIEnableSendfile. The WSGIEnabledSendfile defaults to On in mod_wsgi 3.X, but if mod_wsgi 4.0 ever comes out, it will default to Off there. If it is any file you have issues accessing, not just ones being served up, then check whether you have security features such as SELinux enabled with restrictions on where the Apache user/process may be able access files from. If you are sure that the Apache user can from command line access the files on the NFS file system, then not likely to be a file system permissions issue. Also try using os.listdir() on directories to actually attempt to get file listings from directories and see what happens. Try that first and if no joy I explain another thing you might try to allow you to more interactively work out what the process has access to. Graham On 26/09/2013, at 8:24 PM, regisrob <[email protected]> wrote: > Dear all, > > I desperately come to the community because I'm facing a tricky issue with > mod_wsgi (even if I don't think it's a mod_wsgi bug in itself). I really need > some advice, at least to be sure that my problem is what I think it is, and > if possible have some help to find a solution or workaround. > > ## The issue : it seems that mod_wsgi can't reach data stored on an > NFS-mounted volume > > ## The context : I set up an image server called Loris > (https://github.com/pulibrary/loris) written in Python, that runs behind > Apache using mod_wsgi. Images have to be stored on an NFS-mounted filesystem > > ## My config : > - Debian Squeeze, Linux 2.6.32-5-amd64 > - Python 2.7.3 in virtualenv "/usr/local/loris/bin/python2.7" > - Apache/2.2.16 (Debian) > - mod_wsgi 3.4, configured with /usr/bin/apxs2 and > /usr/local/loris/bin/python2.7 > > The server is working as expected when it delivers images hosted on the main > partition (/dev/sda1). I checked with different locations as source images > root and everything was ok (home/user ; /usr/local/share/images …). But once > I change the source images root to a directory hosted on an NFS mount point > (/mnt/vdata), image requests fail with 404. > > The Loris error caught in Apache log : > 2013-09-25 14:20:49,356 (loris.resolver) [WARNING]: Source image not found at > /mnt/vdata/demos/images/ark_12148_btv1b84473026_T0000006.jp2 for identifier > ark_12148_btv1b84473026_T0000006.jp2. > > sda1 and nfs in /etc/fstab : > # / was on /dev/sda1 during installation > UUID=62dfe91f-8362-413c-ad76-07d348b92ba5 / ext4 > errors=remount-ro 0 1 > 10.10.8.12:/vol/vdata/wbiblissima /mnt/vdata nfs > hard,intr,nolock 0 0 > > > ## My debugging process : > > 1/ check permissions on NFS /mnt/vdata/ => OK, Apache is able to reach > content of all kind from there... > > 2/ check if Python can reach NFS /mnt/vdata/ => OK, using different methods > in python command line : > > >>> import os > >>> print os.path.ismount("/mnt/vdata") > True > > ----------- > > (http://stackoverflow.com/questions/13685239/check-in-python-script-if-nfs-server-is-mounted-and-online) > > from subprocess import Popen, check_output > import time > > def call_timeout(cmd, timeout): > start = time.time() > p = Popen(cmd) > while time.time() - start < timeout: > if p.poll() is not None: > return > time.sleep(0.1) > p.kill() > raise OSError('command timed out') > > call_timeout(["ls", "/mnt/vdata"], 5.0) > > OK for "ls" command > > ----------- > > >>> import os > >>> os.path.isfile("/mnt/vdata/demos/images/ark_12148_btv1b84473026_T0000006.jp2") > True > > > 3/ check if Python can reach NFS /mnt/vdata/ via Apache/mod_wsgi => NOT OK !! > I ran the same tests as above within a sample wsgi app and the output was : > > os.path.exists("/mnt/vdata/") = False > os.path.isfile("/mnt/vdata/demos/images/ark_12148_btv1b84473026_T0000006.jp2") > = False > > But, on the contrary : > > os.path.exists("/usr/local/share/images") = True > os.path.isfile("/usr/local/share/images/ark_12148_btv1b84473026_T0000005.jp2") > = True > > => so it seems pretty sure that Apache/mod_wsgi can't reach NFS-mounted > filesystem (whereas there is no problem with other locations on the main > partition)… > > > 4/ then I read the "Debugging Techniques" page on modwsgi wiki, searched > Google, read some forum threads, but unfortunately nothing concluding… : > > - I tried "EnableSendfile Off" in my virtualhost conf and in apache2.conf (as > explained here > http://pythonhaven.wordpress.com/2009/05/24/configuring-apache-mod_wsgi-django-on-virtual-box-serving-files-from-windows-7/ > ; http://httpd.apache.org/docs/2.2/mod/core.html#enablesendfile) > > - I checked the "middleware wrapper application" given there : > http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Tracking_Request_and_Response > > Here is what i can see in the Apache error logs : > > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] ('REQUEST', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] {'DOCUMENT_ROOT': > '/var/www/vhosts/interne', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'GATEWAY_INTERFACE': > 'CGI/1.1', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_ACCEPT': > 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > 'HTTP_ACCEPT_ENCODING': 'gzip, deflate', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > 'HTTP_ACCEPT_LANGUAGE': 'fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > 'HTTP_CACHE_CONTROL': 'max-age=0', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_CONNECTION': > 'Keep-Alive', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_COOKIE': > '__utma=10294765.500453923.1371827759.1377784603.1377793104.17; > __utmz=10294765.1371827759.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); > Drupal.toolbar.collapsed=0; > _pk_id.1.d439=6077972781a450a8.1373881136.2.1374154272.1373883517.; > Drupal.tableDrag.showWeight=0; > SESS129a620846951a2e9472e600d76a8e6d=0SFg8bdPQFKa0OBrEs4zHmVUs81BnvZfgXtFk2FKx1I', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_HOST': > 'interne.exemple.fr', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_USER_AGENT': > 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:19.0) Gecko/20100101 > Firefox/19.0', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_VIA': '1.1 > interne.exemple.fr', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > 'HTTP_X_FORWARDED_FOR': '191.234.187.220', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > 'HTTP_X_FORWARDED_HOST': 'interne.exemple.fr', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > 'HTTP_X_FORWARDED_SERVER': 'interne.exemple.fr', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'PATH_INFO': > '/ark_12148_btv1b84473026_T0000006.jp2/full/full/0/native.jpg', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'PATH_TRANSLATED': > '/var/www/vhosts/interne/ark_12148_btv1b84473026_T0000006.jp2/full/full/0/native.jpg', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'QUERY_STRING': '', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'REMOTE_ADDR': > '10.14.1.38', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'REMOTE_PORT': > '35296', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'REQUEST_METHOD': > 'GET', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'REQUEST_URI': > '/iiif/ark_12148_btv1b84473026_T0000006.jp2/full/full/0/native.jpg', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SCRIPT_FILENAME': > '/var/www/iiif/loris/loris.wsgi', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SCRIPT_NAME': > '/iiif', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SERVER_ADDR': > '10.14.31.22', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SERVER_ADMIN': > '[email protected]', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SERVER_NAME': > 'interne.exemple.fr', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SERVER_PORT': '80', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SERVER_PROTOCOL': > 'HTTP/1.1', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SERVER_SIGNATURE': > '', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SERVER_SOFTWARE': > 'Apache', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > 'mod_wsgi.application_group': 'interne.exemple.fr|/iiif', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > 'mod_wsgi.callable_object': 'application', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > 'mod_wsgi.enable_sendfile': '0', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > 'mod_wsgi.handler_script': '', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > 'mod_wsgi.input_chunked': '0', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > 'mod_wsgi.listener_host': '', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > 'mod_wsgi.listener_port': '80', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > 'mod_wsgi.process_group': 'loris', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > 'mod_wsgi.queue_start': '1380182347636428', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > 'mod_wsgi.request_handler': 'wsgi-script', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > 'mod_wsgi.script_reloading': '1', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'mod_wsgi.version': > (3, 4), > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'wsgi.errors': > <mod_wsgi.Log object at 0x7f53f460cfb0>, > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'wsgi.file_wrapper': > <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f53f3f62030>, > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'wsgi.input': > <mod_wsgi.Input object at 0x7f53f3fc8370>, > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'wsgi.multiprocess': > True, > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'wsgi.multithread': > True, > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'wsgi.run_once': > False, > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'wsgi.url_scheme': > 'http', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'wsgi.version': (1, > 0)}) > 2013-09-26 09:59:08,734 (loris.resolver) [WARNING]: Source image not found at > /mnt/vdata/demos/images/ark_12148_btv1b84473026_T0000006.jp2 for identifier: > ark_12148_btv1b84473026_T0000006.jp2. > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] ('RESPONSE', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] '404 NOT FOUND', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] [('Content-Type', > 'text/plain; charset=utf-8'), > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] ('Link', > [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] > '<http://library.stanford.edu/iiif/image-api/1.1/compliance.html#level2>;rel="profile"')]) > > When I deploy Loris server with a source images root outside of NFS (e.g. on > /home or /usr/local or anywhere else…), the logs look exactly the same, > except the Response at the end (of course) : > > [Thu Sep 26 10:16:05 2013] [error] [client 10.14.1.38] ('RESPONSE', > [Thu Sep 26 10:16:05 2013] [error] [client 10.14.1.38] '200 OK', > [Thu Sep 26 10:16:05 2013] [error] [client 10.14.1.38] [('Content-Type', > 'image/jpeg'), > [Thu Sep 26 10:16:05 2013] [error] [client 10.14.1.38] ('Link', > [Thu Sep 26 10:16:05 2013] [error] [client 10.14.1.38] > '<http://library.stanford.edu/iiif/image-api/1.1/compliance.html#level2>;rel="profile"'), > [Thu Sep 26 10:16:05 2013] [error] [client 10.14.1.38] ('Last-Modified', > 'Thu, 26 Sep 2013 08:16:05 GMT'), > [Thu Sep 26 10:16:05 2013] [error] [client 10.14.1.38] ('Content-Length', > '2658052')]) > > > Has anyone ever faced that kind of problem before ? What can I do next ? Do > you have a clue for a workaround or another debugging technique ? > > Thank you very much for your help. > > Regis > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/groups/opt_out.
