Dear all, I desperately come to the community because I'm facing a tricky issue with mod_wsgi (even if I don't think it's a mod_wsgi bug in itself). I really need some advice, at least to be sure that my problem is what I think it is, and if possible have some help to find a solution or workaround.
*## The issue* : it seems that mod_wsgi can't reach data stored on an NFS-mounted volume *## The context* : I set up an image server called Loris (https://github.com/pulibrary/loris) written in Python, that runs behind Apache using mod_wsgi. Images have to be stored on an NFS-mounted filesystem *## My config* : - Debian Squeeze, Linux 2.6.32-5-amd64 - Python 2.7.3 in virtualenv "/usr/local/loris/bin/python2.7" - Apache/2.2.16 (Debian) - mod_wsgi 3.4, configured with /usr/bin/apxs2 and /usr/local/loris/bin/python2.7 The server is working as expected when it delivers images hosted on the main partition (/dev/sda1). I checked with different locations as source images root and everything was ok (home/user ; /usr/local/share/images …). But once I change the source images root to a directory hosted on an NFS mount point (/mnt/vdata), image requests fail with 404. The Loris error caught in Apache log : *2013-09-25 14:20:49,356 (loris.resolver) [WARNING]: Source image not found at /mnt/vdata/demos/images/ark_12148_btv1b84473026_T0000006.jp2 for identifier ark_12148_btv1b84473026_T0000006.jp2.* sda1 and nfs in /etc/fstab : *# / was on /dev/sda1 during installation UUID=62dfe91f-8362-413c-ad76-07d348b92ba5 / ext4 errors=remount-ro 0 1 10.10.8.12:/vol/vdata/wbiblissima /mnt/vdata nfs hard,intr,nolock 0 0* *## My debugging process :* *1/* check permissions on NFS /mnt/vdata/ => OK, Apache is able to reach content of all kind from there... *2/* check if Python can reach NFS /mnt/vdata/ => OK, using different methods in python command line : * >>> import os >>> print os.path.ismount("/mnt/vdata") True* ----------- (http://stackoverflow.com/questions/13685239/check-in-python-script-if-nfs-server-is-mounted-and-online) *from subprocess import Popen, check_output import time def call_timeout(cmd, timeout): start = time.time() p = Popen(cmd) while time.time() - start < timeout: if p.poll() is not None: return time.sleep(0.1) p.kill() raise OSError('command timed out') call_timeout(["ls", "/mnt/vdata"], 5.0)* OK for "ls" command ----------- *>>> import os >>> os.path.isfile("/mnt/vdata/demos/images/ark_12148_btv1b84473026_T0000006.jp2") True* *3/* check if Python can reach NFS /mnt/vdata/ via Apache/mod_wsgi => NOT OK !! I ran the same tests as above within a sample wsgi app and the output was : * os.path.exists("/mnt/vdata/") = False os.path.isfile("/mnt/vdata/demos/images/ark_12148_btv1b84473026_T0000006.jp2") = False* But, on the contrary : *os.path.exists("/usr/local/share/images") = True os.path.isfile("/usr/local/share/images/ark_12148_btv1b84473026_T0000005.jp2") = True* => so it seems pretty sure that *Apache/mod_wsgi can't reach NFS-mounted filesystem* (whereas there is no problem with other locations on the main partition)… *4/* then I read the "Debugging Techniques" page on modwsgi wiki, searched Google, read some forum threads, but unfortunately nothing concluding… : - I tried "EnableSendfile Off" in my virtualhost conf and in apache2.conf (as explained here http://pythonhaven.wordpress.com/2009/05/24/configuring-apache-mod_wsgi-django-on-virtual-box-serving-files-from-windows-7/ ; http://httpd.apache.org/docs/2.2/mod/core.html#enablesendfile) - I checked the "middleware wrapper application" given there : http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Tracking_Request_and_Response Here is what i can see in the Apache error logs : [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] ('REQUEST', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] {'DOCUMENT_ROOT': '/var/www/vhosts/interne', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'GATEWAY_INTERFACE': 'CGI/1.1', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_ACCEPT_ENCODING': 'gzip, deflate', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_ACCEPT_LANGUAGE': 'fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_CACHE_CONTROL': 'max-age=0', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_CONNECTION': 'Keep-Alive', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_COOKIE': '__utma=10294765.500453923.1371827759.1377784603.1377793104.17; __utmz=10294765.1371827759.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); Drupal.toolbar.collapsed=0; _pk_id.1.d439=6077972781a450a8.1373881136.2.1374154272.1373883517.; Drupal.tableDrag.showWeight=0; SESS129a620846951a2e9472e600d76a8e6d=0SFg8bdPQFKa0OBrEs4zHmVUs81BnvZfgXtFk2FKx1I', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_HOST': 'interne.exemple.fr', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:19.0) Gecko/20100101 Firefox/19.0', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_VIA': '1.1 interne.exemple.fr', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_X_FORWARDED_FOR': '191.234.187.220', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_X_FORWARDED_HOST': 'interne.exemple.fr', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'HTTP_X_FORWARDED_SERVER': 'interne.exemple.fr', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'PATH_INFO': '/ark_12148_btv1b84473026_T0000006.jp2/full/full/0/native.jpg', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'PATH_TRANSLATED': '/var/www/vhosts/interne/ark_12148_btv1b84473026_T0000006.jp2/full/full/0/native.jpg', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'QUERY_STRING': '', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'REMOTE_ADDR': '10.14.1.38', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'REMOTE_PORT': '35296', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'REQUEST_METHOD': 'GET', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'REQUEST_URI': '/iiif/ark_12148_btv1b84473026_T0000006.jp2/full/full/0/native.jpg', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SCRIPT_FILENAME': '/var/www/iiif/loris/loris.wsgi', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SCRIPT_NAME': '/iiif', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SERVER_ADDR': '10.14.31.22', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SERVER_ADMIN': '[email protected]', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SERVER_NAME': 'interne.exemple.fr', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SERVER_PORT': '80', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SERVER_PROTOCOL': 'HTTP/1.1', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SERVER_SIGNATURE': '', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'SERVER_SOFTWARE': 'Apache', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'mod_wsgi.application_group': 'interne.exemple.fr|/iiif', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'mod_wsgi.callable_object': 'application', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'mod_wsgi.enable_sendfile': '0', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'mod_wsgi.handler_script': '', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'mod_wsgi.input_chunked': '0', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'mod_wsgi.listener_host': '', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'mod_wsgi.listener_port': '80', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'mod_wsgi.process_group': 'loris', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'mod_wsgi.queue_start': '1380182347636428', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'mod_wsgi.request_handler': 'wsgi-script', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'mod_wsgi.script_reloading': '1', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'mod_wsgi.version': (3, 4), [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'wsgi.errors': <mod_wsgi.Log object at 0x7f53f460cfb0>, [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'wsgi.file_wrapper': <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f53f3f62030>, [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'wsgi.input': <mod_wsgi.Input object at 0x7f53f3fc8370>, [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'wsgi.multiprocess': True, [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'wsgi.multithread': True, [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'wsgi.run_once': False, [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'wsgi.url_scheme': 'http', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] 'wsgi.version': (1, 0)}) 2013-09-26 09:59:08,734 (loris.resolver) [WARNING]: Source image not found at /mnt/vdata/demos/images/ark_12148_btv1b84473026_T0000006.jp2 for identifier: ark_12148_btv1b84473026_T0000006.jp2. [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] ('RESPONSE', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] '404 NOT FOUND', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] [('Content-Type', 'text/plain; charset=utf-8'), [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] ('Link', [Thu Sep 26 09:59:08 2013] [error] [client 10.14.1.38] '<http://library.stanford.edu/iiif/image-api/1.1/compliance.html#level2>;rel="profile"')]) When I deploy Loris server with a source images root outside of NFS (e.g. on /home or /usr/local or anywhere else…), the logs look exactly the same, except the Response at the end (of course) : [Thu Sep 26 10:16:05 2013] [error] [client 10.14.1.38] ('RESPONSE', [Thu Sep 26 10:16:05 2013] [error] [client 10.14.1.38] '200 OK', [Thu Sep 26 10:16:05 2013] [error] [client 10.14.1.38] [('Content-Type', 'image/jpeg'), [Thu Sep 26 10:16:05 2013] [error] [client 10.14.1.38] ('Link', [Thu Sep 26 10:16:05 2013] [error] [client 10.14.1.38] '<http://library.stanford.edu/iiif/image-api/1.1/compliance.html#level2>;rel="profile"'), [Thu Sep 26 10:16:05 2013] [error] [client 10.14.1.38] ('Last-Modified', 'Thu, 26 Sep 2013 08:16:05 GMT'), [Thu Sep 26 10:16:05 2013] [error] [client 10.14.1.38] ('Content-Length', '2658052')]) Has anyone ever faced that kind of problem before ? What can I do next ? Do you have a clue for a workaround or another debugging technique ? Thank you very much for your help. Regis -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/groups/opt_out.
