The response to this is complicated. The short answer is that Anaconda Python installations have broken the ability to use their Python distribution in an embedded system such as mod_wsgi. I don’t know that I am fully happy with the workaround I would have to use. If I can’t find a workaround, then only Anaconda can fix the issue, since it appears it is they who have broken the behaviour of the Python C API which is used when doing embedding. On that point they should be fixing it regardless of whether I can find a workaround.
As first step in understanding what is going on, lets first look at running command line Python. When you run command line Python, it will go through a very convoluted algorithm to work out where the Python run time has been installed. >From memory it first looks at the path of the ‘python’ execute run and from >that goes up a directory and then looks for a ‘lib/pythonX.Y’ directory. If the full path to the ‘python’ executable is not known due to how it was run, it will search through $PATH looking for a ‘python’ executable with same name as what was run. From that it again goes looking for the ‘lib/pythonX.Y’ directory. I think it also may even fallback to looking for generic ‘python’ executable name if program name is different. Both of these checks can also be overridden by using the PYTHONHOME environment variable. This would be set to the same directory as ‘sys.prefix’ would give when dumped from the Python interpreter being used. It is very rare situation with the command line Python that one of the first two checks would fail and PYTHONHOME would be required. Now looking at a system where the Python interpreter is embedded in something else, such as is the case with mod_wsgi. With embedded systems, the ‘python’ executable is not run. Instead you are running a custom application which is linked to the Python runtime library. To run the Python interpreter, the custom application would run Py_Initialize() C API function from the Python runtime library. This will in turn trigger the algorithm to find where the Python installation is located. When you talk about an embedded system, this is where problems start to occur. First up, the name of the application is not going to be ‘python’, nor is the path for the application going to be in the same ‘bin’ directory as where Python is located. The mod_wsgi case is even more complicated because the Python library isn’t even linked to the application (Apache httpd) being run. Instead the Python library is linked to an Apache module which is dynamically loaded into Apache only if required. As a result, the first checks will fail as search relative to where ‘httpd' is located will not find a directory ‘lib/python3.5’. Even falling back to looking for ‘python’ executable will fail, as although that will be found in ‘/usr/bin’, that is the system Python which is older version, so again can’t find ‘lib/python3.5’. The only fallback therefore when Python is installed in a non standard location is to tell Python where the Python installation is located. For command line Python you would do that by setting PYTHONHOME environment variable. Setting process environment variables when using Apache is tricky though, also setting it carries some risk. The risk here is that in setting PYTHONHOME that will be inherited by all child processes as well. This means that if somewhere down in your application you wanted to run a Python script which needed to use system Python, overriding PYTHONHOME environment variable could cause that script to fail as it will try and find the Python runtime in the wrong location. The solution for embedded systems is a C API call Py_SetPythonHome(). Rather than set the environment variable, the custom application embedded Python can call this to set the location of the Python installation. This is what the WSGIPythonHome directive is triggering be done. That is, when WSGIPythonHome is set, it will call Py_SetPythonHome() with that value before Py_Initialize() is called so that the Python runtime can be found. And this is now where our problem is. That is, Anaconda Python has somehow broken this C API and calling Py_SetPythonHome() is not overriding where it thinks the Python runtime is located as it should. That said, they aren’t alone in breaking this. Python 3.3+ (???) has also broken Py_SetPythonHome() in the very specific case of where you use the builtin Python pyvenv support. That is, you can’t use Py_SetPythonHome() to refer to a Python virtual environment when you should be able to. This issue has still not been fixed in Python 3 (http://bugs.python.org/issue22213 <http://bugs.python.org/issue22213>). The workaround that mod_wsgi uses when it detects that Python 3 type pyvenv virtual environments are being used is to use Py_SetProgramName() instead. This is hooking into the first check where it tried to find the runtime directory relative to program name. Problem is that for Anaconda you can’t use Py_SetProgramName() either, as they have broken that in some way as well and it doesn’t work either. As far as I can see, using the proper C API for the Python library you cannot make Anaconda work properly for embedded systems in the case where the ‘python’ executable for Anaconda is not in the $PATH for the running program, as it is for Apache, or where PYTHONHOME environment is not being set. If you understand what I am saying here, then you may have picked up that those two things can be used as workarounds. That is, if you can modify the system startup scripts for Apache such that the Anaconda bin directory is in $PATH, then it may work. What will work for sure is setting PYTHONHOME environment variable. So you could modify system startup scripts to set that for Apache, but then you run that risk I mentioned before of causing sub processes that require system Python being screwed up. You might be thinking, if PYTHONHOME works, why I can’t set that inside of mod_wsgi just before Py_Initialize() is called and then unset it after. That would work, but it isn’t as simple as just unsetting it as process environment variable. This is because Python will have already made a copy of it into os.environ of the main interpreter is creates. Any Python code which runs sub process where os.environ is inherited will still get it even if was unset a C level using C unsetenv(). So maybe it could be made to work by using the workaround of setting PYTHONHOME, but does mean that as well as unsetting the environment variable at C level, also need to remove it from os.environ in the main interpreter context as well. I had baulked at such a workaround in the past, but since Anaconda is still causing problems and not even sure that CPython might not further break C APIs for specifying this stuff, may be worth the trouble of seeing if can make it work that way. As to why pyvenv.cfg in the root directory appears to work, that is likely going to be the case as the algorithm to work out where the Python installation is, has from memory some special checks as part of it that allows you to run a compiled Python interpreter out of the source directory where you install it. So it has some checks which look relative to the current directory as well. That pyvenv.cfg check may somehow be related to those checks. This would only work where you can place it in the current working directory of the application when run. The default for Apache for that is the root directory, but not sure that will always work. It will not work with mod_wsgi-express as a workaround and may also have problems with mod_wsgi daemon mode if the home directory option is used. Anyway, that is the long and complicated response. I will have a go at embedding the PYTHONHOME environment variable workaround in mod_wsgi itself and see if that works. It will need to be tested against a range of different Python versions, as well as when using virtualenv and pyvenv though. Additional comments below. > On 6 Sep 2016, at 11:49 PM, Dylan Combs <[email protected]> wrote: > > So, I see that issues similar to this have been raised in the past, but the > issues resolved there do not appear to be the same as the issue as I am > seeing, so hopefully someone can let me know what I'm doing wrong here. I > have developed a workaround using what appears to be undocumented behavior > from the WSGI Module, but I am interested in understanding why seems to be > necessary. > > System Configuration > RHEL 6.8 > httpd 2.2 > mod_wsgi 4.5.6 compiled with: > anaconda 4.1.1 (Python 3.5) > I’m using Red Hat Enterprise Linux 6.8 (fully up to date) with the standard > httpd package delivered and supported for that distribution (so httpd 2.2). > I compiled mod_wsgi from source, so I’m using the latest and greatest (4.5.6) > and I performed that compilation using Python 3.5 through Continuum.io’s > Anaconda software as follows: > > > > > $ wget https://github.com/GrahamDumpleton/mod_wsgi/releases/tag/4.5.6 > $ tar xf 4.5.6.tar && cd 4.5.6 > $ ./configure --with-python=/opt/anaconda3/bin/python3.5 > $ LD_RUN_PATH=/opt/anaconda3/lib make > $ sudo make install > > > The compilation succeeds and the module appears to be properly linked to the > appropriate Python library: > > > > > # ldd /etc/httpd/modules/mod_wsgi.so > linux-vdso.so.1 => (0x00007fffff9f7000) > libpython3.5m.so.1.0 => /opt/anaconda3/lib/libpython3.5m.so.1.0 > (0x00007f6b55b00000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6b558d7000) > libdl.so.2 => /lib64/libdl.so.2 (0x00007f6b556d3000) > libutil.so.1 => /lib64/libutil.so.1 (0x00007f6b554d0000) > librt.so.1 => /lib64/librt.so.1 (0x00007f6b552c7000) > libm.so.6 => /lib64/libm.so.6 (0x00007f6b55043000) > libc.so.6 => /lib64/libc.so.6 (0x00007f6b54caf000) > /lib64/ld-linux-x86-64.so.2 (0x00007f6b56228000) > > > > Initial Trouble > > > > > When I tried to start the httpd daemon using the init script and Upstart > service command provided by RHEL, the attempt failed and I found in > /var/log/httpd/error_log (after increasing too l > evel verbosity): > > > > Could not find platform independent libraries <prefix> > Could not find platform dependent libraries <exec_prefix> > Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>] > [info] mod_wsgi (pid=3786): Python home /opt/anaconda3. > [info] mod_wsgi (pid=3786): Initializing Python. > [info] mod_wsgi (pid=3787): Starting process 'sampleapp' with uid=48, gid=48 > and threads=15. > [info] mod_wsgi (pid=3787): Python home /opt/anaconda3. > [info] mod_wsgi (pid=3787): Initializing Python. > Fatal Python error: Py_Initialize: Unable to get the locale encoding > ImportError: No module named 'encodings' > > And so it begins… > > > > My Quest in Directing Apache to the Anaconda Python Resources > > > > Though it appears from the log and from the documentation on mod_wsgi that I > was correctly establishing the configuration directive necessary to point > httpd to the correct location for Python resources, the error seems pretty > clearly indicative of a failure in that respect. After using every damn WSGI > directive (WSGIPythonHome, WSGIPythonPath, WSGIDaemonProcess with the > python-home or python-path options…) which seemed plausibly related, all to > no avail, I changed tactics. > > > > I ruled out permission and SELinux issues through a variety of standard > means. I tried running Apache as root without SELinux enabled just to make > sure this couldn’t possibly be the problem. Just to be 100% sure and confirm > for myself my own rudimentary assessment capabilities, I even `chmod -R > 777`’d /opt/anaconda3 (not before taking a backup of it which I then restored > after the test, of course). It was clear that httpd could access the files > and do whatever it pleased with them; for some reason, despite seeming to > acknowledge clear direction, even, it just refused to find them. > > > > So I decided I’d just have to stack trace httpd and see what, exactly, is > going on. In doing so, I noticed that the httpd daemon would seem to > successfully locate the /opt/anaconda3 directory (as expected). In > performing the initial module load, it would successfully open the necessary > Python libraries. > > > > open("/opt/anaconda3/lib/libpython3.5m.so.1.0", O_RDONLY) = 5 > > w00t. No issues so far. > > > > However, despite the fact that mod_wsgi logs an info event indicating that it > has properly recognized the WSGIPythonHome directive I am providing (which is > accurate; it is equivalent with the sys.prefix variable, as the documentation > indicates it should be > <https://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIPythonHome.html>), > the startup process invariably devolves into searching what appears to be a > default series of locations for the Python resources (/usr, for example). > > > > From the stack trace (my user name converted to “myusername”): > > > > stat("/sbin/python3", 0x7fffa0fba600) = -1 ENOENT (No such file or > directory) > stat("/bin/python3", 0x7fffa0fba600) = -1 ENOENT (No such file or > directory) > stat("/usr/sbin/python3", 0x7fffa0fba600) = -1 ENOENT (No such file or > directory) > stat("/usr/bin/python3", 0x7fffa0fba600) = -1 ENOENT (No such file or > directory) > readlink("", 0x7fffa0fa5520, 4096) = -1 ENOENT (No such file or > directory) > open("pyvenv.cfg", O_RDONLY) = -1 ENOENT (No such file or > directory) > open("pyvenv.cfg", O_RDONLY) = -1 ENOENT (No such file or > directory) > stat("Modules/Setup", 0x7fffa0fa64a0) = -1 ENOENT (No such file or > directory) > getcwd("/home/myusername", 4096) = 16 > stat("/home/myusername/lib/python3.5/os.py", 0x7fffa0fa64a0) = -1 ENOENT (No > such file or directory) > stat("/home/myusername/lib/python3.5/os.pyc", 0x7fffa0fa63d0) = -1 ENOENT (No > such file or directory) > stat("/home/myusername/lib/python3.5/os.py", 0x7fffa0fa64a0) = -1 ENOENT (No > such file or directory) > stat("/home/myusername/lib/python3.5/os.pyc", 0x7fffa0fa63d0) = -1 ENOENT (No > such file or directory) > stat("/home/lib/python3.5/os.py", 0x7fffa0fa64a0) = -1 ENOENT (No such file > or directory) > stat("/home/lib/python3.5/os.pyc", 0x7fffa0fa63d0) = -1 ENOENT (No such file > or directory) > stat("/home/ilan/minonda/envs/_build/lib/python3.5/os.py", 0x7fffa0fa63d0) = > -1 ENOENT (No such file or directory) > stat("/home/ilan/minonda/envs/_build/lib/python3.5/os.pyc", 0x7fffa0fa6320) = > -1 ENOENT (No such file or directory) > write(2, "Could not find platform independ"..., 55) = 55 > stat("pybuilddir.txt", 0x7fffa0fa1440) = -1 ENOENT (No such file or > directory) > getcwd("/home/myusername", 4096) = 16 > stat("/home/myusername/lib/python3.5/lib-dynload", 0x7fffa0fa5520) = -1 > ENOENT (No such file or directory) > stat("/home/myusername/lib/python3.5/lib-dynload", 0x7fffa0fa5520) = -1 > ENOENT (No such file or directory) > stat("/home/lib/python3.5/lib-dynload", 0x7fffa0fa5520) = -1 ENOENT (No such > file or directory) > stat("/home/ilan/minonda/envs/_build/lib/python3.5/lib-dynload", > 0x7fffa0fa5520) = -1 ENOENT (No such file or directory) > write(2, "Could not find platform dependen"..., 58) = 58 > write(2, "Consider setting $PYTHONHOME to "..., 57) = 57 > > Despite that info event in /var/log/httpd/error_log showing me that mod WSGI > seems to properly recognize the information I provided with the > WSGIPythonHome directive, I never see any attempt from the software to look > to /opt/anaconda3 for the Python resources. If you’ll note, the search > becomes so desperate, in fact, that it includes some weird searches to > subdirectories and resources in the non-existant /home/ilan directory, which > looks like it must be somehow inadvertently included information on behalf of > the designer of Anaconda, as I found a forum post signed “ilan” from him, it > seems. Eventually, as you can see, the search process fails and httpd > proclaims its inability to locate the resources. > > > > I noticed, however, that the search pattern being run by httpd/mod_wsgi looks > like it includes an attempt to locate Python resources using the current > working directory. So… I changed my current working directory to > /opt/anaconda3 and executed httpd manually from there and, voila, if I start > the httpd process from within the Python Home location, it works! GLORY! IT > WORKS! > > > > But… I can’t exactly change the httpd default working directory to solve this > problem; that’s crazy. However, you also might have noticed in my stack > trace that one of the steps Apache was conducting in searching for the Python > resources consisted of attempts to read a pyvenv.cfg file in the current > working directory. > > > > I managed to look up the simple syntax for such a file, and it is something > like this: > > > > home = /opt/anaconda3 > include-system-site-packages = true > version = 3.5.2 > > In fact, it is exactly like that in my case. If I place it in the root of the > file tree (/pyvenv.cfg). That actually fixes the problem and the application > runs as expected without any errors. > > > > The mod_wsgi configuration details I settled on are as follows: > > > > From /etc/httpd/conf.d/awsgi.conf: > > > LoadModule wsgi_module modules/mod_wsgi.so > WSGISocketPrefix /var/run/wsgi That is probably the default directory and prefix for this environment anyway and so WSGISocketPrefix likely wouldn’t have been required. > WSGIApplicationGroup %{GLOBAL} > WSGIPythonHome /opt/anaconda3 > > > And subsequently from /etc/httpd/conf.d/ssl.conf: > > > > <VirtualHost _default_:443> > WSGIScriptAlias / /var/www/wsgi-scripts/scriptname.wsgi > WSGIDaemonProcess sampleapp user=apache group=apache threads=5 The user/group arguments wouldn’t have been needed anyway, as that is what mod_wsgi would default to anyway for this environment. That is, it inherits whatever is used with the User and Group directives in the Apache configuration. > WSGIProcessGroup sampleapp > ... > </VirtualHost> > > > > > Pretty basic. Anyone have any ideas where I went wrong? I am, of course, > glad to attempt any suggestions with the current system and report back on > its behavior. > > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/modwsgi > <https://groups.google.com/group/modwsgi>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
