My problem: I have a large database of interconnected objects which I need to 
process with a combination of short- and long-lived workers. These objects are 
mostly read-only (i.e. any of them can be changed/marked-as-deleted, but that 
happens infrequently). The workers may or may not be within one Python process, 
or even on one system.

I've been doing this with a "classic" session-based SQLAlchemy ORM, approach, 
but that ends up way too slow and memory intense, as each thread gets its own 
copy of every object it needs. I don't want that.

My existing code does object loading and traversal by simple attribute access; 
I'd like to keep that if at all possible.

Ideally, what I'd like to have is an object server which mediates write access 
to the database and then sends change/invalidation notices to the workers. 
(Changes are infrequent enough that I don't care if a worker gets a notice it's 
not interested in.)

I don't care if updates are applied immediately or are only visible to the 
local process until committed. I also don't need fancy indexing or query 
abilities; if necessary I can go to the storage backend for that. (That should 
be SQL, though a NoSQL back-end would be nice to have.)

Does something like this already exist, somewhere out there, or do I need to 
write this, or does somebody know of an alternate solution?

Reply via email to