RE: [Zope3-dev] zodb __parent__ and weakref
Chris, You need to get into the habbit of CC'ing in the list you're replying too... Sorry -- I was thinking that, if this was not a problem w/ ZODB and external objects, but I've missed something in implementation, maybe I should have been communicating on Zope3-users instead. Now I don't know if I've communicated clearly enough to conclude this yet. [EMAIL PROTECTED] wrote: The data is in Postgres, but I already have a python data marshalling/demarshalling/business logic used by other processes, so I want to use this interface with ZOPE as well. Okay, and what is this interface? Do you want to see the *whole* interface ? :) I would think it would suffice to say: it has interfaces to query the database and return python objects constructed from the data, and various business logic interfaces that trigger creation of new data in the database. Zope is only one client. Others will be interacting with the database. Should I be trying to inherit from Persistent rather than __getstate__, __setstate__? ... perhaps there is a way as flagging a member volatile?... Or that I want to control refresh myself? I don't see what implementing IContainer has to do with Persistent, __getstate__ or __setstate__... or am I missing something? I'm only implementing a prototype now, but in production, the database will have =30 million objects in it after 2 years of use. For this reason, and because the db will be changing outside, it doesn't seem sensible to store the db objects in ZODB. This was also the opinion in zope3-users when I asked about this earlier. Instead I want to have a container which queries the database for data itself, but avoids having these objects be put in the zodb, even if it itself is in the zodb. You are saying that it isn't a good approach to implement __getstate__ and __setstate__... but I don't see why this should be so? Can anyone explain why they are called so often? It seemed from other threads that the danger would rather be that your data doesn't get stored at all if you don't explicitly set _p_changed (or whatever its called) I've attached a version of my attempted implementation (cleaned up from the post in [Zope3-Users] newbie design questions for UI to external data. I thought that overriding __getstate__ and __setstate__ simply overrides pythons default versions of these, which would be called just as often as my versions. I guess perhaps the default versions are especially efficient because they just pass the instance dictionary back and forth(? -- but then what does ZODB do with the instance dictionary? Why does it ask for it if the object itself is ok?) Don't worry about any of these things, just don't try and implement anything funky, you don't need to and you're likely to get hurt, as you've found... How should I implement this differently, or what bad assumptions am I making? To my mind, __getstate__ and __setstate__ are the normal way to control persistence in pickle-using python, so funky is in the eye of the beholder. - Shaun Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
RE: [Zope3-dev] zodb __parent__ and weakref
Opps,
The code I said would be attached...
-Original Message-
From: [EMAIL PROTECTED]
[mailto:zope3-dev-
[EMAIL PROTECTED] On Behalf Of Shaun Cutts
Sent: Friday, February 24, 2006 10:18 PM
To: 'Chris Withers'; [EMAIL PROTECTED]; 'zope3-dev'
Subject: RE: [Zope3-dev] zodb __parent__ and weakref
Chris,
You need to get into the habbit of CC'ing in the list you're
replying
too...
Sorry -- I was thinking that, if this was not a problem w/ ZODB and
external objects, but I've missed something in implementation, maybe I
should have been communicating on Zope3-users instead. Now I don't
know
if I've communicated clearly enough to conclude this yet.
[EMAIL PROTECTED] wrote:
The data is in Postgres, but I already have a python data
marshalling/demarshalling/business logic used by other processes, so
I
want to use this interface with ZOPE as well.
Okay, and what is this interface?
Do you want to see the *whole* interface ? :) I would think it would
suffice to say: it has interfaces to query the database and return
python objects constructed from the data, and various business logic
interfaces that trigger creation of new data in the database.
Zope is only one client. Others will be interacting with the database.
Should I be trying to inherit from Persistent rather than
__getstate__,
__setstate__? ... perhaps there is a way as flagging a member
volatile?...
Or that I want to control refresh myself?
I don't see what implementing IContainer has to do with Persistent,
__getstate__ or __setstate__... or am I missing something?
I'm only implementing a prototype now, but in production, the database
will have =30 million objects in it after 2 years of use. For this
reason, and because the db will be changing outside, it doesn't seem
sensible to store the db objects in ZODB. This was also the opinion in
zope3-users when I asked about this earlier.
Instead I want to have a container which queries the database for data
itself, but avoids having these objects be put in the zodb, even if it
itself is in the zodb.
You are saying that it isn't a good approach to implement __getstate__
and __setstate__... but I don't see why this should be so? Can anyone
explain why they are called so often? It seemed from other threads
that
the danger would rather be that your data doesn't get stored at all if
you don't explicitly set _p_changed (or whatever its called)
I've attached a version of my attempted implementation (cleaned up
from
the post in [Zope3-Users] newbie design questions for UI to external
data.
I thought that overriding __getstate__ and __setstate__ simply
overrides
pythons default versions of these, which would be called just as
often
as
my versions. I guess perhaps the default versions are especially
efficient
because they just pass the instance dictionary back and forth(? --
but
then what does ZODB do with the instance dictionary? Why does it ask
for
it if the object itself is ok?)
Don't worry about any of these things, just don't try and implement
anything funky, you don't need to and you're likely to get hurt, as
you've found...
How should I implement this differently, or what bad assumptions am I
making? To my mind, __getstate__ and __setstate__ are the normal way
to
control persistence in pickle-using python, so funky is in the eye
of
the beholder.
- Shaun
Chris
--
Simplistix - Content Management, Zope Python Consulting
- http://www.simplistix.co.uk
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-
dev/shaun%40cuttshome.net
class HollowContainer( IterableUserDict, Contained ):
Implements L{IHollowContainer}
Strategy: for add and delete, notify the database directly; for
update of contained object, however, we want to listen for
ObjectModifiedEvent. This is better than getting a notification
from the objects, as an edit form may set several fields, and then
send the notification when done; so the event can be treated as a commit
Keys: _iserial is key to objects in dictionary
from zope.interface.verify import verifyClass
verifyClass( IHollowContainer, HollowContainer )
True
implements( IHollowContainer, INameChooser )
# defaults for IHollowContainer
database = None
containedInterface = None
# defaults for ILocation (base of IHollowContainer)
__parent__ = None
__name__ = None
def __init__( self ):
super( HollowContainer, self ).__init__( )
self.__setstate__()
def __setitem__( self, key, obj ):
Implements L{cranedata.web.interfaces.IFundContainer.__setitem__}.
if key in self.data:
raise DuplicationError( key %s already present )
self.add( obj )
def add( self, obj ):
obj
Re: [Zope3-dev] zodb __parent__ and weakref
Shaun Cutts wrote: But maybe my I'm misinterpreting the calls to __setstate__? It seems unnecessary to call __setstate__ at all, as we already know the copies in memory are fresh. It sounds like you've implemented both __getstate__ and __setstate__. Why? What's your base concern here, 'cos I don't think you should be worrying about it, whatever it is... Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] zodb __parent__ and weakref
Shaun Cutts wrote: When a container gets pickled to be put into zodb, what happens to the subtree? If the container has references to stuff that is contains, one of two things could happen: 1. If the contents subclass persistent, then they will end up in their own pickle in the zodb and will be made persistent 2. If they don't, they will still be persisted but will end up in the container's pickle in the zodb. Is it just left as a big clump of stuff for the garbage collector to deal with? As with all object, not until nothing references it ;-) If so, maybe __parent__ should be a weakref, Well, if you respect the interface, that's your choice. Not sure a weakref can respect the interface though... because tracing around all of the cycles in the tree by the gc must surely be inefficient, and using a weakref here should make the subtrees dissolve more easily. Why dissolve? cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
RE: [Zope3-dev] zodb __parent__ and weakref
Chris, -Original Message- From: Chris Withers [mailto:[EMAIL PROTECTED] Shaun Cutts wrote: When a container gets pickled to be put into zodb, what happens to the subtree? If the container has references to stuff that is contains, one of two things could happen: 1. If the contents subclass persistent, then they will end up in their own pickle in the zodb and will be made persistent 2. If they don't, they will still be persisted but will end up in the container's pickle in the zodb. Is it just left as a big clump of stuff for the garbage collector to deal with? As with all object, not until nothing references it ;-) What I mean is this: assume that a subtree is *not* referenced anywhere else. It seemed to me that, every time I did a refresh on a container view from the browser, (__getstate__ and then) __setstate__ was called on the container. I would think that means that all the old objects were let go, and new copies were brought in. This would seem to be relatively inefficient to garbage collect that subtree. But it is 2x inefficient to have a tree with lots of cyclic references embedded in it (via __parent__) to garbage collect. However, the situation improves (in the vanilla case where the objects themselves don't have lots of cross references) if __parent__ is a weakref. Then the container, at least, can be immediately deallocated without the garbage collector having to trace the cycles; then the contained objects are also without references, and can be easily garbage collected. Further, it seems that the semantics of the situation dictate that the references from contained.__parent__ to the container should never be the ones keeping the container alive, anyway, as it is part of the tree. But maybe my I'm misinterpreting the calls to __setstate__? It seems unnecessary to call __setstate__ at all, as we already know the copies in memory are fresh. - Shaun ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] zodb __parent__ and weakref
One thing I havent got my mind around yet: I notice that zodb is fairly active about calling __getstate__ and __setstate__. When a container gets pickled to be put into zodb, what happens to the subtree? Is it just left as a big clump of stuff for the garbage collector to deal with? If so, maybe __parent__ should be a weakref, because tracing around all of the cycles in the tree by the gc must surely be inefficient, and using a weakref here should make the subtrees dissolve more easily. - Shaun ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
