My problem: I have a large database of interconnected objects which I need to
process with a combination of short- and long-lived workers. These objects are
mostly read-only (i.e. any of them can be changed/marked-as-deleted, but that
happens infrequently). The workers may or may not be within one Python process,
or even on one system.
I've been doing this with a "classic" session-based SQLAlchemy ORM, approach,
but that ends up way too slow and memory intense, as each thread gets its own
copy of every object it needs. I don't want that.
My existing code does object loading and traversal by simple attribute access;
I'd like to keep that if at all possible.
Ideally, what I'd like to have is an object server which mediates write access
to the database and then sends change/invalidation notices to the workers.
(Changes are infrequent enough that I don't care if a worker gets a notice it's
not interested in.)
I don't care if updates are applied immediately or are only visible to the
local process until committed. I also don't need fancy indexing or query
abilities; if necessary I can go to the storage backend for that. (That should
be SQL, though a NoSQL back-end would be nice to have.)
Does something like this already exist, somewhere out there, or do I need to
write this, or does somebody know of an alternate solution?