What is the external library? I am presuming that 'mathsite' is yours and do not see any evidence in the list of modules you gave me that it in turn imports some third party module wrapping an extension module linking in some third party library.
Graham On 02/03/2015, at 8:34 PM, Paul Royik <[email protected]> wrote: > What if extrnal library takes much memory? > > On Monday, March 2, 2015 at 11:33:48 AM UTC+2, Paul Royik wrote: > OK. > I will try. > > On Monday, March 2, 2015 at 2:26:40 AM UTC+2, Graham Dumpleton wrote: > If I exclude your mathsite modules, on MacOS X at least, importing all the > other modules under mod_wsgi only results in a process size under 12MB. > > What sort of data are your mathsite modules loading as part of the code, or > preloading in some way from somewhere else. For them to take 30MB is quite a > lot. > > Right now because you are preloading mathsite into the task-queue-manager.py > file at global scope, that process itself is quite larger to begin with. At > the least what you should do is organise the algorithm function being called > under the task manager as a wrapper which only lazily imports the mathsite > module when it is called. > > That is, mathsite would not be preloaded in task-queue-manager.py, but would > only be imported after a worker process is forked from it. > > That cuts back the size of the manager process, but you will need to work out > why your mathsite code may be using so much memory even before it does > anything. > > Graham > > On 28/02/2015, at 10:02 PM, Paul Royik <[email protected]> wrote: > > Here you go: > ['Queue', 'UserDict', '__builtin__', '__future__', '__main__', '_abcoll', > '_codecs', '_collections', '_functools', '_hashlib', '_heapq', '_io', > '_locale', '_mod_wsgi_f07fbb7150823bae22c26382ad841b54', '_multiprocessing', > '_random', '_socket', '_sre', '_ssl', '_struct', '_sysconfigdata', > '_warnings', '_weakref', '_weakrefset', 'abc', 'apache', 'array', 'atexit', > 'binascii', 'cPickle', 'cStringIO', 'codecs', 'collections', 'copy_reg', > 'encodings', 'encodings.__builtin__', 'encodings.aliases', 'encodings.ascii', > 'encodings.codecs', 'encodings.encodings', 'encodings.utf_8', 'errno', > 'exceptions', 'fcntl', 'functools', 'gc', 'genericpath', 'hashlib', 'heapq', > 'imp', 'io', 'itertools', 'keyword', 'linecache', 'marshal', 'math', > 'mathsite', 'mathsite.timeout', 'mathsite.timeout.exceptions', > 'mathsite.timeout.mathsite', 'mathsite.timeout.multiprocessing', > 'mathsite.timeout.os', 'mathsite.timeout.signal', 'mathsite.timeout.timeout', > 'mod_wsgi', 'mod_wsgi.pkgutil', 'multiprocessing', 'multiprocessing.Queue', > 'multiprocessing._multiprocessing', 'multiprocessing.array', > 'multiprocessing.atexit', 'multiprocessing.cPickle', > 'multiprocessing.connection', 'multiprocessing.errno', > 'multiprocessing.forking', 'multiprocessing.functools', > 'multiprocessing.itertools', 'multiprocessing.managers', > 'multiprocessing.multiprocessing', 'multiprocessing.os', > 'multiprocessing.pickle', 'multiprocessing.process', > 'multiprocessing.signal', 'multiprocessing.socket', > 'multiprocessing.subprocess', 'multiprocessing.sys', > 'multiprocessing.tempfile', 'multiprocessing.threading', > 'multiprocessing.time', 'multiprocessing.traceback', 'multiprocessing.util', > 'multiprocessing.weakref', 'operator', 'os', 'os.path', 'pickle', 'pkgutil', > 'posix', 'posixpath', 'random', 're', 'select', 'signal', 'site', > 'sitecustomize', 'socket', 'sre_compile', 'sre_constants', 'sre_parse', > 'stat', 'struct', 'subprocess', 'sys', 'sysconfig', 'tempfile', 'thread', > 'threading', 'time', 'traceback', 'types', 'warnings', 'weakref', 'zipimport'] > > On Saturday, February 28, 2015 at 11:36:37 AM UTC+2, Graham Dumpleton wrote: > What code modules does this algorithm require. > > If that is all the (wsgi:service:tasks) processes, then it shows that just > loading all the code and before the process even really does anything is > using 41MB of memory or there about. > > That is quite a lot just for code. You may well still be dragging it too much > stuff you not need. > > That said, there are a lot of Python packages that are really badly > structured and just to use a single function from a module they have a side > effect of dragging in huge amounts of other stuff you do not need in there > parts of the package. > > In the task-queue-manager.py file, just before you you run > 's.serve_forever()', add: > > import sys > print(sorted(sys.modules.keys())) > > Then from Apache error log, get the list of modules you are importing and > post them. > > Graham > > On 28/02/2015, at 12:29 AM, Paul Royik < > ... > > -- > You received this message because you are subscribed to the Google Groups > "modwsgi" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/modwsgi. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
