Thanks for the answer, Armin. The application is corrently using about 15 processes, but this is a configurable number. By your information I can assume that the process overhead would be of about 600MB, which is not a negligible number but still doesn't fully explain all of the memory usage. The application is an information processor, and simplifying the algorithm it would be something like: - start a worker process pool - receive messages from a queue - delegate messages to the workers
The workers themselves have a lifecycle about like that: - get message - fetch data pointed by the message from database (apache cassandra) - process data: - - call a C library to do some preprocessing (Zorba XQuery thru CFFI) - - cal a python module to do intermediate and post processing - store resulting data to database (ElasticSearch) The libraries I'm using are: pycassa, pyelasticsearch, cffi, boto and peewee My operating system is Ubuntu 12.04 LTS x64 Last but not least, is there a way to change pypy from using a GC to using refcount like CPython? That was the information you needed? Feel free to ask anything. Again, thank you, Matheus Salvia 2014-02-03 Armin Rigo <ar...@tunes.org>: > Hi Matheus, > > On 3 February 2014 03:55, Matheus Salvia <matheus2...@gmail.com> wrote: > > When running my app under CPython, it uses about 1GB of memory, but when > > running with pypy it goes up to almost 3GB. > > This is a question that doesn't have a single answer. You need to > give us a lot more information about what your application does, > before we can start proposing ideas about why there is such a > difference in your case. In our own experience, the memory usage is > usually very roughly around the same as CPython (a factor less than 2 > in one way or another). > > As a first (likely completely random) guess, you are starting a very > large number of processes, each of which consumes 60MB instead of 20MB > in CPython. If I'm correct, then there is not much we can do; it's > known that PyPy's baseline memory usage is higher than CPython's, due > to the JIT which ends up storing a few dozens of MBs in any process. > > > A bientôt, > > Armin. > -- -- // Matheus Salvia Desenvolvedor Mobile Celular: +55 11 9-6446-2332 Skype: meta.faraday
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
