On 8 September 2015 at 11:07, Gary Robinson <gary...@me.com> wrote: >> I guess a third possible solution, although it would probably have >> meant developing something for yourself which would have hit the same >> "programmer time is critical" issue that you noted originally, would >> be to create a module that managed the data structure in shared >> memory, and then use that to access the data from the multiple >> processes. > > I think you mean, write a non-python data structure in shared memory, such as > writing it in C? If so, you’re right, I want to avoid the time overhead for > writing something like that. Although I have used C data in shared-memory in > the past when the data structure was simple enough. It’s not a foreign > concept to me — it just would have been a real nuisance in this case. > > An in-memory SQLLite database would have been too slow, at least if I used > any kind of ORM. Without an ORM it still would have slowed things down while > making for code that’s harder to read and write. While I have used in-memory > SQLite code at times, I’m not sure how much slowdown it would have engendered > in this case. > >> Your suggestion (2), of having a non-refcounted data structure is >> essentially this, doable as an extension module. The core data >> structures all use refcounting, and that's unlikely to change, but >> there's nothing to say that an extension module couldn't implement >> fast data structures with objects allocated from a pool of >> preallocated memory which is only freed as a complete block. > > Again, I think you’re talking about non-Python data structures, for instance > C structures, which could be written to be “fast”? Again, I want to avoid > writing that kind of code. Sure, for a production project where I had more > programmer time, that would be a solution, but that wasn’t my situation. And, > ideally, even if I had more time, I would greatly prefer not to have to spend > it on that kind of code. I like Python because it saves me time and > eliminates potential bugs that are associated with language like C but not > with Python (primarily memory management related). To the extent that I have > to write and debug external modules in C or C++, it doesn’t.
I've used cffi to good effect to gain some of the benefits of the "share a lump of memory" model. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
