On 14 Apr 2016 13:04, "Daniel Nicoletti" <[email protected]> wrote: > > Hi, > > yesterday I was doing an experiment to see how fast would it be to uwsgi > spawn 1000 workers with my app loaded, it took ~2s but 1.5GB of RAM, > then instead of 1000 process I started 1 process and 1000 threads, namely > my plugin will instantiate 1000 QThreads, and memory usage was 300MB > now I got surprised of this, so I wrote an application that would load my > app like uwsgi and fork 1000, granted this app doesn't have any protocols > handling, and forking 1000 used 300MB as using threads did. > > uwsgi process here is of 800KB of size and my test app was of 100KB, > which is aproximately the difference. > > Now I wonder why is uwsgi so big? Can it's protocols be split in plugins > loaded at runtime to reduce this? It even seemed that share libraries > didn't add up to free due shared nature, maybe it would be useful if > uwsgi had a shared library, to possibily share among process. Or even > is there something different about how uwsgi forks()? maybe explicity > sharing less if that's even possible? > > Note that I dunno which build options Debian used to uwsgi, maybe it's > possible to compile with reduced size or maybe it would just be better if > it had it's code split into plugins.
You haven't said how you are measuring memory usage. I assume you are measuring it properly and accounting for shared pages, but it would be nice to know for sure. Assuming you are accounting for shared pages properly its worth noting that modern linux malloc is pretty clever, but sometimes does not play nicely with fork and under just the right allocation patterns can result in a memory consolidation process (i forget the details exactly) running just after fork in all children essentially unsharing all the memory the process uses. At my workplace we have observed this before with perl scripts and managed to replicate it with c programs which exactly replicated the allocation sequence the script used. You might want to do the same. Use strace to track all malloc related calls then convert that into a c program which you run and measure. Good luck Yves
_______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
