Thanks! On Nov 29, 7:36 pm, Graham Dumpleton <[email protected]> wrote: > This is always the problem with big packages and can't really suggest much. > > If a package has a lot of Python code, ensuring that it has .pyc files > to facilitate quicker loading is one thing you can do, but I am guess > here that those packages are primarily C extensions and so that > shouldn't be an issue. > > Anyway, this is a question better off asked on the scipy/numpy forms. > > Graham > > On 30 November 2011 06:58, Hang-A <[email protected]> wrote: > > > > > > > > > Hi, Grapham! > > > Thanks for your reference. > > I read it multiple times to understand what is going on. > > > Here is a new problem. > > After setting up the WSGIApplicationGroup directive, > > at least my application runs. > > The new problem is that the speed of importing scipy or numpy is still > > very slow. > > Do you think this problem is still related to the use of the > > simplified GIL api? > > Even in the terminal, it takes a longer time to import scipy or numpy > > on our cluster server running on Cent OS 6. > > > -- Cluster server running on Cent OS 6 > > time python -c 'import scipy' > > real 0m9.734s > > user 0m0.268s > > sys 0m1.188s > > time python -c 'import numpy' > > real 0m3.192s > > user 0m0.169s > > sys 0m0.387s > > > -- my desktop running Mac OS X Leopard > > time python -c 'import scipy' > > real 0m2.385s > > user 0m0.178s > > sys 0m0.260s > > time python -c 'import numpy' > > real 0m0.149s > > user 0m0.081s > > sys 0m0.066s > > > Do you have any ideas about how to increase the speed of scipy/numpy > > import? > > > Thanks! > > > On Nov 7, 1:00 pm, Graham Dumpleton <[email protected]> > > wrote: > >> For some context as to why it helps see: > > >>http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Python_Simpli... > > >> Graham > > >> On 8 November 2011 03:39, Hang-A <[email protected]> wrote: > > >> > That was it. > >> > Thanks, Graham! > > >> > Myung-Hwa > > >> > On Nov 5, 5:42 pm, Graham Dumpleton <[email protected]> > >> > wrote: > >> >> Try setting: > > >> >> WSGIApplicationGroup %{GLOBAL} > > >> >> for that WSGI application in Apache configuration. > > >> >> Likely you are using a third party C extension module in there > >> >> somewhere which isn't safe to use in sub interpreter. > > >> >> That directive will force your application to run in main Python > >> >> interpreter within the process. > > >> >> Graham > > >> >> On 6 November 2011 06:56, Hang-A <[email protected]> wrote: > > >> >> > Hello, list! > > >> >> > First of all, this will be a long message, due to the complexity of > >> >> > our environment. > >> >> > So, please be patient with my question. > > >> >> > I am trying to run a simple django application in a cluster > >> >> > environment. > >> >> > And, my application hangs while it imports scipy.linalg, and both > >> >> > scipy and apache do not write out error messages. > >> >> > When I run my application in my local python shell, it imports > >> >> > scipy.linalg. But, somehow it does not when it is run by apache. > >> >> > So, after reading this message, please share any ideas about how to > >> >> > debug this problem or new solutions to address this issue or deploy my > >> >> > application. > > >> >> > Now, let me explain our current setup. > >> >> > 1. OS > >> >> > -- The server is a compute cluster where each node runs centos 6 that > >> >> > was installed from a clean version of centos6 minimal. > >> >> > 2. Apache > >> >> > -- Apache 2.2 was also manually installed from one of default linux > >> >> > repository. > >> >> > To be specific, it was installed from its source code together with > >> >> > httpd-dev. > >> >> > 3. Python > >> >> > -- Python 2.7.2 was also installed from its source code across all > >> >> > nodes in the cluster. Its source code was downloaded from python.org's > >> >> > ftp. > >> >> > 4. Python packages: nose, numpy, scipy > >> >> > -- Nose 1.1.2 was downloaded from pypi.python.org and installed from > >> >> > its source code. > >> >> > -- numpy 1.6.1 was downloaded and installed from a linux repository. > >> >> > When building numpy, gnu95 fortran complier was used. > >> >> > -- To install scipy, we installed atlas-3.8.4, lapack-3.3.1, and blas > >> >> > from their source code. > >> >> > ----- atlas was from sourceforge's 3.8.4 stable version. To compile > >> >> > altas, gcc was used. > >> >> > ----- lapack and blas was obtained from netlib.org's repository. To > >> >> > compile the package of lapack and blas, gforan was used. > >> >> > ----- Finally, after exporting paths to blas, lapack, and atlas, > >> >> > scipy-0.9.0 was installed from its source code. > >> >> > scipy was obtained from sourceforge.net's repository. > > >> >> > All of the above were installed in the same way across all nodes in > >> >> > our cluster. > >> >> > Since I am the only user of the cluster who needs to run python web > >> >> > applications, > >> >> > I installed python virtualenv package in my local directory. > >> >> > Within my virtual environment, django-1.3 and pysal-1.2 (our own > >> >> > package) were installed. > >> >> > To deploy my web applications, we used mod_wsgi. > >> >> > mod-wsgi was compiled with python-2.7.2 and loaded into apache-2.2. > > >> >> > My application is as follows: > >> >> > import os, sys, site, time > >> >> > print >> sys.stderr, str(time.time()) + ':' + str(time.ctime()) > > >> >> > root_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), > >> >> > '..')) > >> >> > root_dir += '/myunghwa_python2' > >> >> > print >> sys.stderr, root_dir > >> >> > print >> sys.stderr, os.listdir(root_dir) > >> >> > site.addsitedir(os.path.join(root_dir, 'lib/python2.7/site-packages')) > >> >> > sys.path.append(root_dir) > >> >> > print >> sys.stderr, sys.path > >> >> > print >> sys.stderr, os.listdir(sys.path[-2]) > > >> >> > os.environ['PYTHON_EGG_CACHE'] = '/home/mhwang3/repos/django/python- > >> >> > eggs' > >> >> > os.environ['DJANGO_SETTINGS_MODULE'] = 'wsgi.settings' > > >> >> > def application(environ, start_response): > >> >> > status = '200 OK' > >> >> > output = 'Hello!' > >> >> > try: > >> >> > t1 = time.time() > >> >> > print >> sys.stderr, "Going to import numpy: " + > >> >> > str(time.time()) + ":" + str(time.time() - t1) > >> >> > import numpy > >> >> > print >> sys.stderr, "imported numpy" > >> >> > print >> sys.stderr, "Going to import scipy" > >> >> > import scipy > >> >> > print >> sys.stderr, "imported scipy > >> >> > from:"+str(scipy.__path__) > >> >> > print >> sys.stderr, "Going to import scipy.linalg" + > >> >> > str(time.time()) + ':' + str(time.time() - t1) > >> >> > import scipy.stats > >> >> > print >> sys.stderr, "imported scipy.linalg: " + > >> >> > str(time.time()) + ':' + str(time.time() - t1) > >> >> > print >> sys.stderr, "Going to import pysal" > >> >> > import pysal > >> >> > print >> sys.stderr, "imported pysal" > >> >> > except: > >> >> > print >> sys.stderr, "exception" > >> >> > raise > > >> >> > response_headers = [('Content-type', 'text/plain'), > >> >> > ('Content-Length', str(len(output)))] > >> >> > start_response(status, response_headers) > > >> >> > return [output] > > >> >> > Basically, it is a 'hello world' application that tests if numpy, > >> >> > scipy, and pysal can be imported. > >> >> > In the attached file, lines 4-9 are just adding paths to django and > >> >> > pysal so that apache knows where to find these packages. > >> >> > Also, to let apache know where to find atlas-related packages, the > >> >> > path to those packages was added to the LD_LIBRARY_PATH environment > >> >> > variable in the /etc/sysconfig/httpd file. > > >> >> > When I first ran my application, it just hung and wrote no message. > >> >> > So, across scipy.linalg modules, I added print out statements to > >> >> > figure out at which point the import was broken. > >> >> > Here is the messages I got when I imported scipy.linalg in my local > >> >> > python shell. > >> >> > ######################## > >> >> > starting linalg.__init__ > >> >> > pre __init__.__doc__ > >> >> > pre __init__.__version__ > >> >> > pre __init__.misc > >> >> > pre __init__.basic > >> >> > ####################### > >> >> > Starting basic > >> >> > pre basic.flinalg > >> >> > pre basic.lapack > >> >> > pre basic.misc > >> >> > pre basic.scipy.linalg > >> >> > pre basic.decomp_svd > >> >> > pre __init__.decomp > >> >> > ################ > >> >> > starting decomp > >> >> > pre decomp.array et al. > >> >> > pre decomp.calc_lwork > >> >> > pre decomp.LinAlgError > >> >> > pre decomp.get_lapack_funcs > >> >> > pre decomp.get_blas_funcs > >> >> > #################### > >> >> > Starting blas > >> >> > pre blas.scipy.linalg.fblas > >> >> > pre blas.scipy.linalg.cblas > >> >> > pre __init__.decomp_lu > >> >> > pre __init__.decomp_cholesky > >> >> > pre __init__.decomp_qr > >> >> > ################# > >> >> > Starting special_matrices > >> >> > pre special_matrices.math > >> >> > pre special_matrices.np > >> >> > pre __init__.decomp_svd > >> >> > pre __init__.decomp_schur > >> >> > ################## > >> >> > starting schur... > >> >> > pre decomp_schur.misc > >> >> > pre decomp_schur.LinAlgError > >> >> > pre decomp_schur.get_lapack_funcs > >> >> > pre decomp_schur.eigvals:1320454147.23Fri Nov 4 17:49:07 2011 > >> >> > schur testing > >> >> > pre __init__.matfuncs > >> >> > ##################### > >> >> > Starting matfuncs > >> >> > pre matfuncs. asarray et al > >> >> > pre matfuncs.matrix > >> >> > pre matfuncs.np > >> >> > pre matfuncs.misc > >> >> > pre matfuncs.basic > >> >> > pre matfuncs.special_matrices > >> >> > pre matfuncs.decomp > >> >> > pre matfuncs.decomp_svd > >> >> > pre matfuncs.decomp_schur > >> >> > pre __init__.blas > >> >> > pre __init__.special_matrices > >> >> > When scipy.linalg is successfully imported, I should get these > >> >> > messages. > >> >> > But, when my web application tried to import scipy.linalg, the output > >> >> > messages stop at line 41. > >> >> > At line 41, decomp_schur.py tries to import decomp.py. Since decomp.py > >> >> > was already imported at line 16, scipy ignores it and continues to > >> >> > import > > ... > > read more »
-- You received this message because you are subscribed to the Google Groups "modwsgi" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
