OK. I will try. On Monday, March 2, 2015 at 2:26:40 AM UTC+2, Graham Dumpleton wrote: > > If I exclude your mathsite modules, on MacOS X at least, importing all the > other modules under mod_wsgi only results in a process size under 12MB. > > What sort of data are your mathsite modules loading as part of the code, > or preloading in some way from somewhere else. For them to take 30MB is > quite a lot. > > Right now because you are preloading mathsite into the > task-queue-manager.py file at global scope, that process itself is quite > larger to begin with. At the least what you should do is organise the > algorithm function being called under the task manager as a wrapper which > only lazily imports the mathsite module when it is called. > > That is, mathsite would not be preloaded in task-queue-manager.py, but > would only be imported after a worker process is forked from it. > > That cuts back the size of the manager process, but you will need to work > out why your mathsite code may be using so much memory even before it does > anything. > > Graham > > On 28/02/2015, at 10:02 PM, Paul Royik <[email protected] <javascript:>> > wrote: > > Here you go: > ['Queue', 'UserDict', '__builtin__', '__future__', '__main__', '_abcoll', > '_codecs', '_collections', '_functools', '_hashlib', '_heapq', '_io', > '_locale', '_mod_wsgi_f07fbb7150823bae22c26382ad841b54', > '_multiprocessing', '_random', '_socket', '_sre', '_ssl', '_struct', > '_sysconfigdata', '_warnings', '_weakref', '_weakrefset', 'abc', 'apache', > 'array', 'atexit', 'binascii', 'cPickle', 'cStringIO', 'codecs', > 'collections', 'copy_reg', 'encodings', 'encodings.__builtin__', > 'encodings.aliases', 'encodings.ascii', 'encodings.codecs', > 'encodings.encodings', 'encodings.utf_8', 'errno', 'exceptions', 'fcntl', > 'functools', 'gc', 'genericpath', 'hashlib', 'heapq', 'imp', 'io', > 'itertools', 'keyword', 'linecache', 'marshal', 'math', 'mathsite', > 'mathsite.timeout', 'mathsite.timeout.exceptions', > 'mathsite.timeout.mathsite', 'mathsite.timeout.multiprocessing', > 'mathsite.timeout.os', 'mathsite.timeout.signal', > 'mathsite.timeout.timeout', 'mod_wsgi', 'mod_wsgi.pkgutil', > 'multiprocessing', 'multiprocessing.Queue', > 'multiprocessing._multiprocessing', 'multiprocessing.array', > 'multiprocessing.atexit', 'multiprocessing.cPickle', > 'multiprocessing.connection', 'multiprocessing.errno', > 'multiprocessing.forking', 'multiprocessing.functools', > 'multiprocessing.itertools', 'multiprocessing.managers', > 'multiprocessing.multiprocessing', 'multiprocessing.os', > 'multiprocessing.pickle', 'multiprocessing.process', > 'multiprocessing.signal', 'multiprocessing.socket', > 'multiprocessing.subprocess', 'multiprocessing.sys', > 'multiprocessing.tempfile', 'multiprocessing.threading', > 'multiprocessing.time', 'multiprocessing.traceback', > 'multiprocessing.util', 'multiprocessing.weakref', 'operator', 'os', > 'os.path', 'pickle', 'pkgutil', 'posix', 'posixpath', 'random', 're', > 'select', 'signal', 'site', 'sitecustomize', 'socket', 'sre_compile', > 'sre_constants', 'sre_parse', 'stat', 'struct', 'subprocess', 'sys', > 'sysconfig', 'tempfile', 'thread', 'threading', 'time', 'traceback', > 'types', 'warnings', 'weakref', 'zipimport'] > > On Saturday, February 28, 2015 at 11:36:37 AM UTC+2, Graham Dumpleton > wrote: > > What code modules does this algorithm require. > > If that is all the (wsgi:service:tasks) processes, then it shows that just > loading all the code and before the process even really does anything is > using 41MB of memory or there about. > > That is quite a lot just for code. You may well still be dragging it too > much stuff you not need. > > That said, there are a lot of Python packages that are really badly > structured and just to use a single function from a module they have a side > effect of dragging in huge amounts of other stuff you do not need in there > parts of the package. > > In the task-queue-manager.py file, just before you you run > 's.serve_forever()', add: > > import sys > print(sorted(sys.modules.keys())) > > Then from Apache error log, get the list of modules you are importing and > post them. > > Graham > > On 28/02/2015, at 12:29 AM, Paul Royik < > > ...
-- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
