Graham,
I get the following when i run module-config. Using this only I copied the
wsgi module path.
LoadFile "/Users/vishnuhari/anaconda/lib/libpython2.7.dylib"
LoadModule wsgi_module
"/Users/vishnuhari/anaconda/lib/python2.7/site-packages/mod_wsgi/server/mod_wsgi-py27.so"
WSGIPythonHome "/Users/vishnuhari/anaconda"
Further if you look at vhost entry below..am giving all three paths in
python-home
<VirtualHost *:80>
ServerAdmin [email protected]
ServerName www.groc-example.com
ServerAlias groc-example.com
WSGIScriptAlias /images/ /var/www/images/wsgiimagehandler.py
ErrorLog "/private/var/log/apache2/image_error_log"
CustomLog "/private/var/log/apache2/image_custom_log" common
WSGIDaemonProcess images python-home=/users/vishnuhari/anaconda
python-path=/var/www/images/ processes=1 threads=10 inactivity-timeout=30
maximum-requests=20
WSGIProcessGroup images
<Directory "/var/www/images/">
Options +ExecCGI
Allow from all
Require all granted
</Directory>
</VirtualHost>
Note that in my code when I do run the simple server, my make server is as
follows which is running on 8051. I have simplifed the entire code below
for brevity. The actual code can be seen
@ https://github.com/vishnuprasadh/imgprocessor/blob/master/wsgiimagehandler.py
#!usr/bin/env python
from wsgiref.simple_server import make_server
from wsgiref.handlers import CGIHandler
import io
from mod_wsgi import server
from imagehandler import imagehandler
import cgi
def application(environment, start_response):
returnquerystring = ""
response_body = "Oops,we are sorry!"
status = '200 OK'
try:
if environment["PATH_INFO"] == "/images/":
img = imagehandler()
returnquerystring = img.generate()
'''In the response header, we will also set imagekey for use in varnish
and other edgecache servers'''
response_headers = [
('Content-Type', 'image/jpeg'),
# ("Content-Type","text/html"),
('Content-Length', str(len(response_body))),
('Server', "Encrypt"),
('imagekey', str(returnquerystring))]
except Exception as genex:
status = '400 Bad Request'
response_body = str(genex)
response_headers = [
# ('Content-Type', 'image/jpeg'),
("Content-Type", "text/html"),
('Content-Length', str(len(response_body))),
('Server', "Encrypt"),
('Response_QS', str(returnquerystring))]
finally:
start_response(status, response_headers)
return [response_body]
'''Make the server -note that the requests shoudl come to this port'''
httpd = make_server('', 8051, application)
'''Continue to listen on this server forever'''
httpd.serve_forever()
The error I get is :
[Tue Apr 11 13:07:04.618145 2017] [wsgi:error] [pid 3217] [remote
127.0.0.1:55330] File
"/users/vishnuhari/anaconda/lib/python2.7/mimetools.py", line 6, in <module>
[Tue Apr 11 13:07:04.618475 2017] [wsgi:error] [pid 3217] [remote
127.0.0.1:55330] import tempfile
[Tue Apr 11 13:07:04.618495 2017] [wsgi:error] [pid 3217] [remote
127.0.0.1:55330] File
"/users/vishnuhari/anaconda/lib/python2.7/tempfile.py", line 32, in <module>
[Tue Apr 11 13:07:04.619091 2017] [wsgi:error] [pid 3217] [remote
127.0.0.1:55330] import io as _io
[Tue Apr 11 13:07:04.619109 2017] [wsgi:error] [pid 3217] [remote
127.0.0.1:55330] File "/users/vishnuhari/anaconda/lib/python2.7/io.py",
line 51, in <module>
[Tue Apr 11 13:07:04.619416 2017] [wsgi:error] [pid 3217] [remote
127.0.0.1:55330] import _io
[Tue Apr 11 13:07:04.619444 2017] [wsgi:error] [pid 3217] [remote
127.0.0.1:55330] ImportError:
dlopen(/users/vishnuhari/anaconda/lib/python2.7/lib-dynload/_io.so, 2):
Symbol not found: __PyCodecInfo_GetIncrementalDecoder
[Tue Apr 11 13:07:04.619449 2017] [wsgi:error] [pid 3217] [remote
127.0.0.1:55330] Referenced from:
/users/vishnuhari/anaconda/lib/python2.7/lib-dynload/_io.so
[Tue Apr 11 13:07:04.619452 2017] [wsgi:error] [pid 3217] [remote
127.0.0.1:55330] Expected in: flat namespace
[Tue Apr 11 13:07:04.619455 2017] [wsgi:error] [pid 3217] [remote
127.0.0.1:55330] in
/users/vishnuhari/anaconda/lib/python2.7/lib-dynload/_io.so
Note that this is currently working if i run the script through python
standalone bound with nohup. It is also working through
mod_wsgi-express....
First step is to resolve this to run out of core apache2 outside embedded.
Again, my worry is in production setup..I dont have such luxury as in my
local. How do I simplify/streamline this entire thing because ultimately
this will run in a cluster of apache servers sitting on RHEL 6.x.
thanks in anticipation...
On Saturday, April 8, 2017 at 9:29:16 AM UTC+5:30, Graham Dumpleton wrote:
>
> On this issue, the problem is likely that mod_wsgi is picking up the wrong
> Python library because Anaconda Python is in a non standard location and
> you have a system Python of same major/minor version.
>
> When you configure/make/make install mod_wsgi from source code against
> Anaconda Python, you need to set LD_RUN_PATH to the library directory for
> Anaconda Python so it knows where to find the correct library at run time.
>
> Thus:
>
> make distclean
>
> ./configure --python=/users/vishnuhari/anaconda/bin/python
>
> LD_RUN_PATH=/users/vishnuhari/anaconda/lib make
> LD_RUN_PATH=/users/vishnuhari/anaconda/lib make install
>
> Alternatively, you need to work out path for the Anaconda Python library
> and add a line into Apache configuration before the LoadModule, which says:
>
> LoadFile /users/vishnuhari/anaconda/lib/libpython2.7.so
>
> The path make be slightly different depending on name given to the library
> by Anaconda Python. Ie., could be:
>
> LoadFile /users/vishnuhari/anaconda/lib/libpython2.7m.so
>
> If you ran:
>
> mod_wsgi-express module-config
>
> it should tell you the correct lines, including the LoadFile line, to put
> in the Apache configuration.
>
> Send an email with what you get from output of the mod_wsgi-express
> module-config command.
>
> Graham
>
> On 8 Apr 2017, at 3:09 AM, Vishnu Prasad <[email protected]
> <javascript:>> wrote:
>
> I am trying to move away from embedded and configuring the apache2 (not
> mod_wsgi embedded), virtual hosts but keep getting this issue of Symbol Not
> found , Expected in flat namespace.
>
> I added following virtualhost config.
> <VirtualHost *:80>
> ServerAdmin [email protected] <javascript:>
> ServerName www.groc-example.com
> ServerAlias groc-example.com
> WSGIScriptAlias /images/ /var/www/images/wsgiimagehandler.py
> ErrorLog "/private/var/log/apache2/image_error_log"
> CustomLog "/private/var/log/apache2/image_custom_log" common
> WSGIDaemonProcess images python-home=/users/vishnu/anaconda
> python-path=/var/www/images/ processes=1 threads=10 inactivity-timeout=30
> maximum-requests=20
> WSGIProcessGroup images
> WSGIApplicationGroup %{GLOBAL}
> <Directory "/var/www/images/">
> Options +ExecCGI
> Allow from all
> Require all granted
> </Directory>
> </VirtualHost>
>
>
> I added LoadModule wsgi_module
> "/Users/vishnuhari/anaconda/lib/python2.7/site-packages/mod_wsgi/server/mod_wsgi-py27.so"
> in
> the httpd.conf.
>
> Also uncommented the line which has following
> LoadModule cgi_module libexec/apache2/mod_cgi.so
> Include /private/etc/apache2/extra/httpd-vhosts.conf
>
>
> After i did tha above, have restarted and start apachectl,
>
> ERROR :
>
> [Fri Apr 07 22:26:25.838308 2017] [wsgi:error] [pid 1634] [remote
> 127.0.0.1:54121] File
> "/users/vishnu/anaconda/lib/python2.7/mimetools.py", line 6, in <module>
> [Fri Apr 07 22:26:25.838389 2017] [wsgi:error] [pid 1634] [remote
> 127.0.0.1:54121] import tempfile
> [Fri Apr 07 22:26:25.838398 2017] [wsgi:error] [pid 1634] [remote
> 127.0.0.1:54121] File
> "/users/vishnu/anaconda/lib/python2.7/tempfile.py", line 32, in <module>
> [Fri Apr 07 22:26:25.838535 2017] [wsgi:error] [pid 1634] [remote
> 127.0.0.1:54121] import io as _io
> [Fri Apr 07 22:26:25.838544 2017] [wsgi:error] [pid 1634] [remote
> 127.0.0.1:54121] File "/users/vishnu/anaconda/lib/python2.7/io.py",
> line 51, in <module>
> [Fri Apr 07 22:26:25.838615 2017] [wsgi:error] [pid 1634] [remote
> 127.0.0.1:54121] import _io
> [Fri Apr 07 22:26:25.838635 2017] [wsgi:error] [pid 1634] [remote
> 127.0.0.1:54121] ImportError:
> dlopen(/users/vishnuhari/anaconda/lib/python2.7/lib-dynload/_io.so, 2):
> Symbol not found: __PyCodecInfo_GetIncrementalDecoder
> [Fri Apr 07 22:26:25.838640 2017] [wsgi:error] [pid 1634] [remote
> 127.0.0.1:54121] Referenced from:
> /users/vishnuhari/anaconda/lib/python2.7/lib-dynload/_io.so
> [Fri Apr 07 22:26:25.838643 2017] [wsgi:error] [pid 1634] [remote
> 127.0.0.1:54121] Expected in: flat namespace
> [Fri Apr 07 22:26:25.838646 2017] [wsgi:error] [pid 1634] [remote
> 127.0.0.1:54121] in
> /users/vishnuhari/anaconda/lib/python2.7/lib-dynload/_io.so
>
>
>
>
>
> On Friday, April 7, 2017 at 1:05:26 PM UTC+5:30, Vishnu Prasad wrote:
>>
>> Graham, Thanks a ton for the response.
>>
>> I have updated more inputs inline for your review and suggestions.
>>
>> thanks
>>
>> On Friday, April 7, 2017 at 8:57:22 AM UTC+5:30, Graham Dumpleton wrote:
>>>
>>>
>>> On 7 Apr 2017, at 9:40 AM, Vishnu Prasad <[email protected]> wrote:
>>>
>>> All,
>>> I am new to mod_wsgi and the entire configurations around related to
>>> this. Detailing out the requirement, please do let me know your response.
>>>
>>> 1. I have written a simple python script to handle images which listens
>>> on port 8041. There is some custom processing happening here and its not
>>> just static rendering. The output of this is a content of type image and
>>> bytes is the response.
>>> 2. I am starting this python script file(application) using command
>>> mod_wsgi-express start-server abc.py --threads 20 --process 2
>>>
>>>
>>> Are you doing image processing? If you are and it is CPU intensive, then
>>> use of threads, rather than processes, isn't then best idea.
>>>
>> [Vishnu:] Yes, am using Pillow-SIMD to dynamically generate or
>> resize images for optimize image rendering. Once returned from here,
>> caching in varnish. We have over 100,000 items to process and at any given
>> point easily and i will need to generate about 20-30 images/sec. This
>> webserver cluster is about 4 and sitting behind LB. I have done image
>> domain randomization to leverage capability of browsers to download more
>> parallel images. Created about 3 subdomains for this as well, hence
>> processes of 2. Agree - Once performance tested we may tune this further.
>> Further this webserver is dedicated to passthrough request to next level
>> appserver outside of which this is the only CPU intense process working on
>> these.
>>
>>>
>>> 3. I have varnish script that whenever gets any requests for images on
>>> 8080 routes to this application on port 8041.
>>> 4. This application is also running another default instance of apache2
>>> which is serving my all other application requests not related to images.
>>>
>>>
>>> And this is a Python application also hosted on mod_wsgi, but configured
>>> manually in normal Apache instance rather than using mod_wsgi-express?
>>> [Vishnu:] Yes, this is on mod_wsgi but not configured through normal
>>> apache instance. As you said in one of your blog, getting it working was
>>> first - so started in embedded mode and now reviewing if I should move out
>>> or keep it that way for one set of URL's. This means all images goes to
>>> mod_wsgi-express and others go to the normal apache default instance.
>>> Earlier I thought of having a virtualhost config to route this for all
>>> /image patterns i.e. Daemon config as you said in one of the blog, later
>>> realized about the modularity of this setup. I wanted to use this as
>>> stand-alone modular imageserver. I am reading through your link below on
>>> the issues with embedded mode of running. I will review both these options
>>> and run some performance tests. This is helpful.
>>>
>>> The above setup is working.
>>>
>>> Problem/Help needed:
>>> 1. I want to configure and run the above application as distinct
>>> WSGIDaemonProcess. How do I achieve this ?
>>>
>>>
>>> If for (4) you are running a Python application hosted on mod_wsgi, you
>>> should already be using daemon mode as embedded node is generally not
>>> recommended as explained in:
>>>
>>>
>>> http://blog.dscpl.com.au/2012/10/why-are-you-using-embedded-mode-of.html
>>> <http://www.google.com/url?q=http%3A%2F%2Fblog.dscpl.com.au%2F2012%2F10%2Fwhy-are-you-using-embedded-mode-of.html&sa=D&sntz=1&usg=AFQjCNGiURKOSUfshe-g4r-68VMGRnLOYQ>
>>>
>> [Vishnu:] I am running this app in embedded mode as i was planning to use
>> this more to manage/handle only image requests.
>>
>>>
>>> 2. If the above ask is accomplished by a configuration in default
>>> apache httpd.conf (not mod_wsgi httpd.conf) then i also want to know how
>>> will apache identify my application vs other applications of mod_wsgi which
>>> may be running ? Is it based on server url or ?
>>>
>>>
>>> How to vertically partition a URL namespace so as to directory subsets
>>> of URLs to a different daemon process group within the same Apache
>>> instance, is explained in:
>>>
>>>
>>> http://blog.dscpl.com.au/2014/02/vertically-partitioning-python-web.html
>>> <http://www.google.com/url?q=http%3A%2F%2Fblog.dscpl.com.au%2F2014%2F02%2Fvertically-partitioning-python-web.html&sa=D&sntz=1&usg=AFQjCNFSqPozGdQVlEtWMQUYSzWo1LEzOQ>
>>>
>>> 3. For the above application, need to have better control over what is
>>> loaded in httpd.conf. So, want to load my own httpd.conf including path of
>>> log mounts etc. How do I achieve this ?
>>>
>>> [Vishnu:] My question was how do i specifically setup a httpd.conf for
>>> this embedded mode i.e. outside the core apache2 installation. Based on
>>> what i read from you its better to not run in embedded mode. However,
>>> if we need to then the configuration around logs for this specifically will
>>> need to happen in mod_wsgi hence the ask.
>>>
>> Again, am not generating apache config instead am asking how do i tell
>> mod_wsgi-express, instead of you reading from /tmp/....httpd.conf, read
>> this from /usr/local/httpd.conf ? Is there some server-base path setting
>> which i could use and set this or ? I completely understand your point and
>> dont want to mess up over-engineering the solution.
>>
>>>
>>> I don't understand what you mean by this.
>>>
>>> You should not be using mod_wsgi-express to generate Apache
>>> configuration to then use in an existing Apache installation. The
>>> mod_wsgi-express instance is self contained with a generated configuration.
>>> If customisations for that are required, most common things can be
>>> overridden using the command line options. For some special cases you can
>>> use --include-file to add additional configuration at the end of the
>>> generated configuration if necessary.
>>>
>>
>>
>>
>> 4. Finally, at somepoint if I want to manage the httpd.conf common file
>>> across WSGI and my default apache, how do I set up the application specific
>>> configurations - through virtualhost section in httpd-vhost.conf or ?
>>>
>>>
>>> If for the other (4) above the answer is that you are also using a
>>> manually configure Apache/mod_wsgi, as well as a separate mod_wsgi-express
>>> for just one subset of URLs, then as per blog on vertically partition, you
>>> probably should just do it all manually in one Apache/mod_wsgi instance if
>>> want to have config all in one spot. Personally I would probably use
>>> multiple mod_wsgi-express instance sitting directly behind varnish or
>>> whatever front end you are using. The generated configuration from
>>> mod_wsgi-express is usually going to be better than what many would set up
>>> manually as it sets a whole lot of good stuff related to timeouts which
>>> makes things more robust. When Apache/mod_wsgi is manually configured there
>>> are no default timeouts set.
>>>
>> [Vishnu:] Understand that you suggest against embdedded mode and against
>> managing 2 apache conf files. Even if we need, then you suggest me to take
>> just mod_wsgi-express. I will review this as said earlier as well. Also, i
>> went through the httpd.conf of mod_wsgi process for WSGIDaemon node and its
>> quite extensive. I will keep you posted on the outcome. thanks a ton for a
>> detailed answer, the links are helpful and am reading through them for
>> guidance.
>>
>>>
>>> Anyway, suggest you read that blog on vertically partition a web
>>> application at least and clarify things by answering questions above.
>>>
>>> Graham
>>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected] <javascript:>.
> To post to this group, send email to [email protected] <javascript:>
> .
> Visit this group at https://groups.google.com/group/modwsgi.
> For more options, visit https://groups.google.com/d/optout.
>
>
>
--
You received this message because you are subscribed to the Google Groups
"modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
