i'm also working on zope/plone integration though at a different layer
then florent, i'm focusing more on mapping to the plone application
schema model, then the zodb object storage. to be opensourced, probably
another 2 weeks.
diving in to hand out 2 cents ;-)
Michael Bayer wrote:
On Feb 27, 2006, at 6:17 AM, Florent Guillaume wrote:
1. I'd like the possibility of having the identity map not cleared on
commit. This turns identity maps into proper cross-requests caches in
the context of web applications. This is what ZODB does, and it's
extremely beneficial to have non-modified objects kept in caches
associated to a pool.
its not cleared on commit. its only cleared if you do
objectstore.clear(). if you are using just commits, or explicit
begin/commits, the identity map stays in place.
unfortunately the problem is cache coherency in the identity map, my
understanding (not sure) is that when being used by multiple threads as
in zope, each thread would have its own session and identity map, while
it may be self consistent with changes in that thread, changes in
another thread would mean the identity map would be inconsistent with
the database state, without invalidations across identity maps. unless
i've misunderstood something..
2. I'd like to replace the identity map implementation with another
one derived from Zope's ZODB (the persistent.PickleCache, coded in C).
If you really wanted this, Id provide some kind of hook into objectstore
where alternate implementations can take place. But I dont think this
is what youre looking for. a picklecache sounds like its serializing
objects and putting them inside of a file, is it cross-process
accessibility youre looking for ? keep in mind the identity map is not
an object cache, thats not its purpose at all....its strictly to
maintain the identity of objects as unique within a particular session.
if youre looking for caching that would be something else.
PickleCache has a number of very interesting features, most notably
automatic size management with a target maximum size, the rest may be
garbage collected (LRU eviction).
now I know you are talking about a cache. this is not approrpriate for
an identity map...it must maintain a reference to all objects that are
in memory, else duplicate instances can occur. the current identity
map is a WeakValueDictionary so no additional objects are stored that
are not already present in an application's in-memory namespace, so it
incurs no memory overhead at all right now (other than weakref
containers) (maybe thats why you thought it clears out after a commit?).
if it will work it still seems pretty ideal.
the pickle cache does much the same as the identity map plus with
bounding properties about the cache size, in that its keeps a memory
reference to objects, and it never evicts objects till explicitly asked
to do so typically at transaction boundaries, and would first changes
the object to the ghost state and switches to a weak reference as a
second class citizen in the cache, since the ghost doesn't have by
definition any state its eviction is safe, since any attempt to access
an in memory copy of the ghost would wake it up and switch it to back to
a strong reference in the cache.
some pickle cache links for more reading (pretty old, but probably the
best out there ~2002).
http://www.zope.org/Members/htrd/cache/impl
http://www.zope.org/Members/htrd/cache
and the implementation which has some notes on how it works
http://svn.zope.org/ZODB/trunk/src/persistent/cPickleCache.c?rev=29896&view=auto
the pickle cache is used by the zodb along with cross cache invalidation
to keep a persistent in memory objectstore.
4. Another interesting feature of the persistent.Persistent class is
the notion of "ghost". A ghost is a stub for an object that's lazily
loaded. It has a very small memory footprint. On any attribute access,
its manager (_p_jar) is called to fetch the object's state.
Conversely, when the PickleCache is purged by the LRU mechanism, old
entries are not deleted but turned back into ghosts.
this could be built on top of an SA-managed set of classes and probably
could be provided as an extension option. not totally sure how this is
used ? if I ask the mapper "select all objects where x=12", it hits the
DB anyway, is it just that you dont want to fetch the full column set ?
deferred properties can already be used for this ? (i.e. you can have
all object attributes load as "deferred" in a single lazy query,
http://www.sqlalchemy.org/docs/adv_datamapping.myt#adv_datamapping_deferred
)
i think the nearest analogy in sa terms would be an sa mapped object
which just held its primary key, until any of its attributes were looked
at which point it would load the non deferred ones.
this notion is built into the pickle cache, ie deactivation of objects
not in use to conserve memory.
i think for most usages the deferred property will suffice, and for the
purposes of cache integration the mapped object's class could have
support for it, treating the object's primary key as a _p_oid (zodb
object unique reference).
5. I'll have to add two-phase commit to SA because this is essential
to Zope's transaction mechanism (which may manage several databases).
> is this compatible with DBAPI ? if you can show me how DBAPI is used
> within a two-phase commit we can begin to think how such a feature might
> be implemented. if the whole notion occurs externally to DBAPI, its
> possible that you would need only provide a custom implementation of the
> Session to provide this (its worth looking at the source code to the
> Session and UnitOfWork objects in objectstore.py, they are pretty
> simple. the supporting objects used during an actual commit to sort
> dependencies, i.e.UOWTransaction on down, much more complicated, dont
> read too far down in the module :) ...maybe I should break those out
> into a separate module.)
>
i was able to add this in pretty easily in an engine subclass, i null
implemented the existing sa transaction machinery and let zope's txn
manager drive.
more on the zope transaction interface
http://svn.zope.org/ZODB/trunk/src/transaction/interfaces.py?rev=41164&view=auto
cheers,
kapil
-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
Sqlalchemy-users mailing list
Sqlalchemy-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlalchemy-users
