On Thu, Aug 27, 2009 at 7:48 AM, Stuart Bishop <[email protected]>wrote:
> On Thu, Aug 27, 2009 at 4:56 PM, Gustavo Niemeyer<[email protected]> > wrote: > > Hey Stuart, > > > >> Has anyone looked into making Storm objects pickleable? I want to > >> stuff expensive query results into memcached. > >> > >> I'm using ZStorm so can just use the name to refer to the Store. I can > >> put together a MaterializedResultSet class supporting a lot of the API > >> from a materialized list of Storm objects. I think getting the Storm > >> objects themselves pickled is going to be the tricky bit. > > > > Pickling itself shouldn't be hard. How do you envison an unpickled > > object should behave? > > Its nice to know you don't forsee major roadblocks. I think the major > difficulty is becoming familiar enough with the Storm internals - I've > never dealt with ObjectInfo and friends before. > > I'd like it to behave like the original object as much as possible. > The goal is drop in replacement of code like: > > results = store.find(... complex and costly conditions ...) > > with something like: > > results = cached(store, max_age, ... complex and costly conditions ...) > > So unpickled objects can be updated and code able to traverse from the > unpickled objects to objects loaded from the Store. For the Storm > objects, I expect they would be indistinguishable from one loaded from > the Store (assemble object, swap in the Store, inject into the cache). > > I don't think I need the result set to support operations like union, > find, or aggregates beyond count() so the result set can just be a > list with a count() method. > i have slightly different use cases for caching, i'd like to be able to cache these independent of a store. effectively separate from store, pickle, unpickle, and attach to store. analogous to sqlalchemy session detach/merge functionality. afaics this involves clearing out the instance object info, and possibly unhooking the event system from tracking variable changes. cheers, kapil
-- storm mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/storm
