What if extrnal library takes much memory? On Monday, March 2, 2015 at 11:33:48 AM UTC+2, Paul Royik wrote: > > OK. > I will try. > > On Monday, March 2, 2015 at 2:26:40 AM UTC+2, Graham Dumpleton wrote: >> >> If I exclude your mathsite modules, on MacOS X at least, importing all >> the other modules under mod_wsgi only results in a process size under 12MB. >> >> What sort of data are your mathsite modules loading as part of the code, >> or preloading in some way from somewhere else. For them to take 30MB is >> quite a lot. >> >> Right now because you are preloading mathsite into the >> task-queue-manager.py file at global scope, that process itself is quite >> larger to begin with. At the least what you should do is organise the >> algorithm function being called under the task manager as a wrapper which >> only lazily imports the mathsite module when it is called. >> >> That is, mathsite would not be preloaded in task-queue-manager.py, but >> would only be imported after a worker process is forked from it. >> >> That cuts back the size of the manager process, but you will need to work >> out why your mathsite code may be using so much memory even before it does >> anything. >> >> Graham >> >> On 28/02/2015, at 10:02 PM, Paul Royik <[email protected]> wrote: >> >> Here you go: >> ['Queue', 'UserDict', '__builtin__', '__future__', '__main__', '_abcoll', >> '_codecs', '_collections', '_functools', '_hashlib', '_heapq', '_io', >> '_locale', '_mod_wsgi_f07fbb7150823bae22c26382ad841b54', >> '_multiprocessing', '_random', '_socket', '_sre', '_ssl', '_struct', >> '_sysconfigdata', '_warnings', '_weakref', '_weakrefset', 'abc', 'apache', >> 'array', 'atexit', 'binascii', 'cPickle', 'cStringIO', 'codecs', >> 'collections', 'copy_reg', 'encodings', 'encodings.__builtin__', >> 'encodings.aliases', 'encodings.ascii', 'encodings.codecs', >> 'encodings.encodings', 'encodings.utf_8', 'errno', 'exceptions', 'fcntl', >> 'functools', 'gc', 'genericpath', 'hashlib', 'heapq', 'imp', 'io', >> 'itertools', 'keyword', 'linecache', 'marshal', 'math', 'mathsite', >> 'mathsite.timeout', 'mathsite.timeout.exceptions', >> 'mathsite.timeout.mathsite', 'mathsite.timeout.multiprocessing', >> 'mathsite.timeout.os', 'mathsite.timeout.signal', >> 'mathsite.timeout.timeout', 'mod_wsgi', 'mod_wsgi.pkgutil', >> 'multiprocessing', 'multiprocessing.Queue', >> 'multiprocessing._multiprocessing', 'multiprocessing.array', >> 'multiprocessing.atexit', 'multiprocessing.cPickle', >> 'multiprocessing.connection', 'multiprocessing.errno', >> 'multiprocessing.forking', 'multiprocessing.functools', >> 'multiprocessing.itertools', 'multiprocessing.managers', >> 'multiprocessing.multiprocessing', 'multiprocessing.os', >> 'multiprocessing.pickle', 'multiprocessing.process', >> 'multiprocessing.signal', 'multiprocessing.socket', >> 'multiprocessing.subprocess', 'multiprocessing.sys', >> 'multiprocessing.tempfile', 'multiprocessing.threading', >> 'multiprocessing.time', 'multiprocessing.traceback', >> 'multiprocessing.util', 'multiprocessing.weakref', 'operator', 'os', >> 'os.path', 'pickle', 'pkgutil', 'posix', 'posixpath', 'random', 're', >> 'select', 'signal', 'site', 'sitecustomize', 'socket', 'sre_compile', >> 'sre_constants', 'sre_parse', 'stat', 'struct', 'subprocess', 'sys', >> 'sysconfig', 'tempfile', 'thread', 'threading', 'time', 'traceback', >> 'types', 'warnings', 'weakref', 'zipimport'] >> >> On Saturday, February 28, 2015 at 11:36:37 AM UTC+2, Graham Dumpleton >> wrote: >> >> What code modules does this algorithm require. >> >> If that is all the (wsgi:service:tasks) processes, then it shows that >> just loading all the code and before the process even really does anything >> is using 41MB of memory or there about. >> >> That is quite a lot just for code. You may well still be dragging it too >> much stuff you not need. >> >> That said, there are a lot of Python packages that are really badly >> structured and just to use a single function from a module they have a side >> effect of dragging in huge amounts of other stuff you do not need in there >> parts of the package. >> >> In the task-queue-manager.py file, just before you you run >> 's.serve_forever()', add: >> >> import sys >> print(sorted(sys.modules.keys())) >> >> Then from Apache error log, get the list of modules you are importing and >> post them. >> >> Graham >> >> On 28/02/2015, at 12:29 AM, Paul Royik < >> >> ... > >
-- You received this message because you are subscribed to the Google Groups "modwsgi" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/modwsgi. For more options, visit https://groups.google.com/d/optout.
