Hi, Grapham! Thanks for your reference. I read it multiple times to understand what is going on.
Here is a new problem. After setting up the WSGIApplicationGroup directive, at least my application runs. The new problem is that the speed of importing scipy or numpy is still very slow. Do you think this problem is still related to the use of the simplified GIL api? Even in the terminal, it takes a longer time to import scipy or numpy on our cluster server running on Cent OS 6. -- Cluster server running on Cent OS 6 time python -c 'import scipy' real 0m9.734s user 0m0.268s sys 0m1.188s time python -c 'import numpy' real 0m3.192s user 0m0.169s sys 0m0.387s -- my desktop running Mac OS X Leopard time python -c 'import scipy' real 0m2.385s user 0m0.178s sys 0m0.260s time python -c 'import numpy' real 0m0.149s user 0m0.081s sys 0m0.066s Do you have any ideas about how to increase the speed of scipy/numpy import? Thanks! On Nov 7, 1:00 pm, Graham Dumpleton <[email protected]> wrote: > For some context as to why it helps see: > > http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Python_Simpli... > > Graham > > On 8 November 2011 03:39, Hang-A <[email protected]> wrote: > > > > > > > > > That was it. > > Thanks, Graham! > > > Myung-Hwa > > > On Nov 5, 5:42 pm, Graham Dumpleton <[email protected]> > > wrote: > >> Try setting: > > >> WSGIApplicationGroup %{GLOBAL} > > >> for that WSGI application in Apache configuration. > > >> Likely you are using a third party C extension module in there > >> somewhere which isn't safe to use in sub interpreter. > > >> That directive will force your application to run in main Python > >> interpreter within the process. > > >> Graham > > >> On 6 November 2011 06:56, Hang-A <[email protected]> wrote: > > >> > Hello, list! > > >> > First of all, this will be a long message, due to the complexity of > >> > our environment. > >> > So, please be patient with my question. > > >> > I am trying to run a simple django application in a cluster > >> > environment. > >> > And, my application hangs while it imports scipy.linalg, and both > >> > scipy and apache do not write out error messages. > >> > When I run my application in my local python shell, it imports > >> > scipy.linalg. But, somehow it does not when it is run by apache. > >> > So, after reading this message, please share any ideas about how to > >> > debug this problem or new solutions to address this issue or deploy my > >> > application. > > >> > Now, let me explain our current setup. > >> > 1. OS > >> > -- The server is a compute cluster where each node runs centos 6 that > >> > was installed from a clean version of centos6 minimal. > >> > 2. Apache > >> > -- Apache 2.2 was also manually installed from one of default linux > >> > repository. > >> > To be specific, it was installed from its source code together with > >> > httpd-dev. > >> > 3. Python > >> > -- Python 2.7.2 was also installed from its source code across all > >> > nodes in the cluster. Its source code was downloaded from python.org's > >> > ftp. > >> > 4. Python packages: nose, numpy, scipy > >> > -- Nose 1.1.2 was downloaded from pypi.python.org and installed from > >> > its source code. > >> > -- numpy 1.6.1 was downloaded and installed from a linux repository. > >> > When building numpy, gnu95 fortran complier was used. > >> > -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas > >> > from their source code. > >> > ----- atlas was from sourceforge's 3.8.4 stable version. To compile > >> > altas, gcc was used. > >> > ----- lapack and blas was obtained from netlib.org's repository. To > >> > compile the package of lapack and blas, gforan was used. > >> > ----- Finally, after exporting paths to blas, lapack, and atlas, > >> > scipy-0.9.0 was installed from its source code. > >> > scipy was obtained from sourceforge.net's repository. > > >> > All of the above were installed in the same way across all nodes in > >> > our cluster. > >> > Since I am the only user of the cluster who needs to run python web > >> > applications, > >> > I installed python virtualenv package in my local directory. > >> > Within my virtual environment, django-1.3 and pysal-1.2 (our own > >> > package) were installed. > >> > To deploy my web applications, we used mod_wsgi. > >> > mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2. > > >> > My application is as follows: > >> > import os, sys, site, time > >> > print >> sys.stderr, str(time.time()) + ':' + str(time.ctime()) > > >> > root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), > >> > '..')) > >> > root_dir += '/myunghwa_python2' > >> > print >> sys.stderr, root_dir > >> > print >> sys.stderr, os.listdir(root_dir) > >> > site.addsitedir(os.path.join(root_dir, 'lib/python2.7/site-packages')) > >> > sys.path.append(root_dir) > >> > print >> sys.stderr, sys.path > >> > print >> sys.stderr, os.listdir(sys.path[-2]) > > >> > os.environ['PYTHON_EGG_CACHE'] = '/home/mhwang3/repos/django/python- > >> > eggs' > >> > os.environ['DJANGO_SETTINGS_MODULE'] = 'wsgi.settings' > > >> > def application(environ, start_response): > >> > status = '200 OK' > >> > output = 'Hello!' > >> > try: > >> > t1 = time.time() > >> > print >> sys.stderr, "Going to import numpy: " + > >> > str(time.time()) + ":" + str(time.time() - t1) > >> > import numpy > >> > print >> sys.stderr, "imported numpy" > >> > print >> sys.stderr, "Going to import scipy" > >> > import scipy > >> > print >> sys.stderr, "imported scipy > >> > from:"+str(scipy.__path__) > >> > print >> sys.stderr, "Going to import scipy.linalg" + > >> > str(time.time()) + ':' + str(time.time() - t1) > >> > import scipy.stats > >> > print >> sys.stderr, "imported scipy.linalg: " + > >> > str(time.time()) + ':' + str(time.time() - t1) > >> > print >> sys.stderr, "Going to import pysal" > >> > import pysal > >> > print >> sys.stderr, "imported pysal" > >> > except: > >> > print >> sys.stderr, "exception" > >> > raise > > >> > response_headers = [('Content-type', 'text/plain'), > >> > ('Content-Length', str(len(output)))] > >> > start_response(status, response_headers) > > >> > return [output] > > >> > Basically, it is a 'hello world' application that tests if numpy, > >> > scipy, and pysal can be imported. > >> > In the attached file, lines 4-9 are just adding paths to django and > >> > pysal so that apache knows where to find these packages. > >> > Also, to let apache know where to find atlas-related packages, the > >> > path to those packages was added to the LD_LIBRARY_PATH environment > >> > variable in the /etc/sysconfig/httpd file. > > >> > When I first ran my application, it just hung and wrote no message. > >> > So, across scipy.linalg modules, I added print out statements to > >> > figure out at which point the import was broken. > >> > Here is the messages I got when I imported scipy.linalg in my local > >> > python shell. > >> > ######################## > >> > starting linalg.__init__ > >> > pre __init__.__doc__ > >> > pre __init__.__version__ > >> > pre __init__.misc > >> > pre __init__.basic > >> > ####################### > >> > Starting basic > >> > pre basic.flinalg > >> > pre basic.lapack > >> > pre basic.misc > >> > pre basic.scipy.linalg > >> > pre basic.decomp_svd > >> > pre __init__.decomp > >> > ################ > >> > starting decomp > >> > pre decomp.array et al. > >> > pre decomp.calc_lwork > >> > pre decomp.LinAlgError > >> > pre decomp.get_lapack_funcs > >> > pre decomp.get_blas_funcs > >> > #################### > >> > Starting blas > >> > pre blas.scipy.linalg.fblas > >> > pre blas.scipy.linalg.cblas > >> > pre __init__.decomp_lu > >> > pre __init__.decomp_cholesky > >> > pre __init__.decomp_qr > >> > ################# > >> > Starting special_matrices > >> > pre special_matrices.math > >> > pre special_matrices.np > >> > pre __init__.decomp_svd > >> > pre __init__.decomp_schur > >> > ################## > >> > starting schur... > >> > pre decomp_schur.misc > >> > pre decomp_schur.LinAlgError > >> > pre decomp_schur.get_lapack_funcs > >> > pre decomp_schur.eigvals:1320454147.23Fri Nov 4 17:49:07 2011 > >> > schur testing > >> > pre __init__.matfuncs > >> > ##################### > >> > Starting matfuncs > >> > pre matfuncs. asarray et al > >> > pre matfuncs.matrix > >> > pre matfuncs.np > >> > pre matfuncs.misc > >> > pre matfuncs.basic > >> > pre matfuncs.special_matrices > >> > pre matfuncs.decomp > >> > pre matfuncs.decomp_svd > >> > pre matfuncs.decomp_schur > >> > pre __init__.blas > >> > pre __init__.special_matrices > >> > When scipy.linalg is successfully imported, I should get these > >> > messages. > >> > But, when my web application tried to import scipy.linalg, the output > >> > messages stop at line 41. > >> > At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py > >> > was already imported at line 16, scipy ignores it and continues to > >> > import other modules in my local shell. > >> > But, somehow, in apache-mod_wsgi environment, scipy failed to ignore > >> > or reload decomp.py and seems to kill my web application. > >> > This is really odd, because python does not give any message about > >> > this error and neither does apache. apache just hangs without sending > >> > out any response. > >> > Since lapack and blas functions were imported successfully, the > >> > problem seems not related to path setup. > > >> > If anyone in the list has any insights into or experience into this > >> > kind of symptom, > >> > please share your insights and experience. In particular, debugging > >> > techniques or less-known installation/compilation problems would be > >> > helpful. > >> > I feel like I am at a dead end. So, please help me. > > >> > Thanks for reading this post. > >> > I will look forward to your responses. > > >> > -- Myunghwa-Hwang Hwang > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups "modwsgi" group. > >> > To post to this group, send email to [email protected]. > >> > To unsubscribe from this group, send email to > >> > [email protected]. > >> > For more options, visit this group > >> > athttp://groups.google.com/group/modwsgi?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "modwsgi" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/modwsgi?hl=en. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
