Hi Rob, Which version of PyPy are you running this with? I have a long running branch that I really should merge someday that is supposed to help with memory consumption of json deserialization. Is there a chance you could share a (anonymized) version of your test file?
Alternatively, you could try a nightly build from this branch yourself: http://buildbot.pypy.org/nightly/json-decoder-maps-py3.6/ Carl Friedrich On August 22, 2019 1:02:42 AM GMT+02:00, Robert Whitcher <robert.whitc...@rubrik.com> wrote: >Hi, >I am running a very simple test case (as we are hitting OOM on our >larger >PyPy deployments) and I'd love some help understanding what is >happening >here.... >We have a lot of processes that send messages to each other. >These can be large JSON serializations of objects. >But the memory being consumed seems out of order and hard to manage >across >processes. > >I have this loop running: > >import time >import json > >def main(): > with open("/tmp/test12334.1234", "r") as f: > json_msg = f.read() > > while True: > j = json.loads(json_msg) > time.sleep(10) > >if __name__ == "__main__": > main() > > >I have tried 3 separate general runs across both pypy and cpython. >The first does nothing but the sleep (it doesn't load or json the >message) >The second just loaded the json_str from a file >The third is the full loop. > >If I run this in cpython I get (80MB, 92MB and 136MB) respectively >This makes sense as the file is 11MB json serialization of a dictionary >and >json.loads takes up some memory > >However if I run this in pypy I get 120MB, 153MB and between 360-405MB >when >it settles out. >I get the JIT and startup memory being higher, spending a little more >loading the string but WOW does json loading the string chew up a >bunch. > >Multiplying that memory across processes is eating a bunch. > >What easy things am I missing? > >Thanks, >Rob
_______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
