Holy grail for me (use case being IPC) would be:
Python process 1 mmap's a file shared with Python process 2.
Python process 1 puts a string into mmap'ed file, pointer to that is
"somehow transferred" to process 2 (eg pushing the index into the mmap'ed
file over Unix domain socket .. a single uint64), and Python code in
process 2 can do stuff with this string _without_ copying - probably via
zero_buffer.
Have you actually measured that copying the data between processes is the
bottleneck? Using shared memory is something I tried also for a client of
mine but at the end we switched back to pass messages using network because
the extra complexity was not worth the gain. But again, I suppose it
depends on the size of the message.
I have measured (and profiled using vmprof) that we have bottlenecks in
serialization/deserialization .. the overhead is lowest (on PyPy) when
using a pure CBOR implementation, but still.
Hence my interest in captnproto/flatbuffers.
I also benchmarked Unix domain socket pure bandwidth (in C) .. which is
very high indeed (UDS only involves a single copy when done in C, not 2
copies like loopback TCP).
I probably should stop worrying/guessing and try capnpy over UDS and see
how far this will get us;)
Cheers,
/Tobias
ciao,
Anto
_______________________________________________
pypy-dev mailing list
pypy-dev@python.org
https://mail.python.org/mailman/listinfo/pypy-dev
