On Thu, Aug 27, 2009 at 8:28 AM, Kapil Thangavelu <[email protected]>wrote:
> > > On Thu, Aug 27, 2009 at 7:48 AM, Stuart Bishop < > [email protected]> wrote: > >> On Thu, Aug 27, 2009 at 4:56 PM, Gustavo Niemeyer<[email protected]> >> wrote: >> > Hey Stuart, >> > >> >> Has anyone looked into making Storm objects pickleable? I want to >> >> stuff expensive query results into memcached. >> >> >> >> I'm using ZStorm so can just use the name to refer to the Store. I can >> >> put together a MaterializedResultSet class supporting a lot of the API >> >> from a materialized list of Storm objects. I think getting the Storm >> >> objects themselves pickled is going to be the tricky bit. >> > >> > Pickling itself shouldn't be hard. How do you envison an unpickled >> > object should behave? >> >> Its nice to know you don't forsee major roadblocks. I think the major >> difficulty is becoming familiar enough with the Storm internals - I've >> never dealt with ObjectInfo and friends before. >> >> I'd like it to behave like the original object as much as possible. >> The goal is drop in replacement of code like: >> >> results = store.find(... complex and costly conditions ...) >> >> with something like: >> >> results = cached(store, max_age, ... complex and costly conditions ...) >> >> So unpickled objects can be updated and code able to traverse from the >> unpickled objects to objects loaded from the Store. For the Storm >> objects, I expect they would be indistinguishable from one loaded from >> the Store (assemble object, swap in the Store, inject into the cache). >> >> I don't think I need the result set to support operations like union, >> find, or aggregates beyond count() so the result set can just be a >> list with a count() method. >> > > > i have slightly different use cases for caching, i'd like to be able to > cache these independent of a store. > effectively separate from store, pickle, unpickle, and attach to store. > analogous to sqlalchemy session > detach/merge functionality. > > afaics this involves clearing out the instance object info, and possibly > unhooking the event system from tracking variable changes. > > cheers, > > kapil > i ended up making a base class to mimic the store merge/detach functionality along with the pickling support (inline below) that i needed. modifying the object when its detached is effectively an unknown op as there isn't a store to track it, although upon reattaching any subsequent modification and flushes will include the modifications while detached. hasn't been tested/used with references. cheers, kapil from storm import properties, info marker = object() class StormPersistent( object ): def __getstate__( self ): d = {} for p,v in info.get_obj_info(self).variables.items(): d[p.name] = v.get() return d def __setstate__( self, o ): for p,v in info.get_obj_info( self ).variables.items(): value = o.get( p.name, marker ) if value is marker: continue v.set( value, from_db=True ) def detach( self ): obj_info = info.get_obj_info( self ) store = obj_info.get('store') assert not obj_info in store._dirty, "Can't Detach Dirty Object" store._remove_from_alive( obj_info ) store._disable_change_notification( obj_info ) store._disable_lazy_resolving( obj_info ) def attach( self, store ): obj_info = info.get_obj_info( self ) obj_info['store'] = store store._enable_change_notification( obj_info ) store._add_to_alive( obj_info ) store._enable_lazy_resolving( obj_info )
-- storm mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/storm
