Re: [Zope3-Users] What attributes are made persistent
Peter Bengtsson wrote: So by using PersistentList it just means that you can use: self.alist.append(1) in your code. The attribute, self.alist, is still saved even without PersistentList but it just means you have to be careful when writing to it. PersistentX classes also usually subclass Persistent which means they get their own pickle jar. This can slow ZODB growth if you have lots of big attributes but where only one of them regularly changes: make the changing one a PersistentWhatever instance. cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] What attributes are made persistent
On 2/16/06, Jeff Shell [EMAIL PROTECTED] wrote: One could, but it's really not worth it. It's just the laws of Python and mutability and immutability :). (It took me years to understand those terms. I kept associating them with 'mutable' in the can be made quiet sense... Eventually my music brain stepped back and I went oh, MUTATE! Ahhh!. Seriously, it took me about four years to understand that :). Wow! I think I love you Jeff. Not only do you reply with non-condencending constructive advise you also confess to [past] weaknesses. Personally, I seldom store lists, or even use them as attributes on instances. Even outside of Zope/ZODB, I've found myself accidentally losing data because I was carelessly using the reference. Tuples for everybody! Ok. Sounds sensible. I've learnt something: use BTrees() instead of dict(), use tuple() instead of list(). On 2/15/06, Shaun Cutts [EMAIL PROTECTED] wrote: Well, one could have a base class along the lines of class PersistSetItemOnAttributes: def __setattr__( self, attr, val ): oldSI = val.__class__.__dict__.get( '__setitem__', None ) if oldSI is not None: def newSI( vself, vattr, vval ): vself._p_changed = True # is this the right member? oldSI( vattr, vval )# oldSI is bound: no vself? val.__class__.__setitem__ = newSI super( PersistSetItemOnAttributes, self ).__setattr__( attr, val ) Of course, this is really just pseudocode. For one thing, need to trap whether 'val' is really a class. For another if you were serious about this, you would want to check if the obj wasn't persistent first, and you might want to do it recursively. And while your are at it, you might want to check on other mutators as well(for instance, check first what sequence interface if any 'val' supports...) ... sounds like too much work, and would be problem prone even so. After all, some things you don't want to be persistent! - Shaun -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Stephan Richter Sent: Wednesday, February 15, 2006 8:43 AM To: zope3-users@zope.org Cc: Florian Lindner Subject: Re: [Zope3-Users] What attributes are made persistent On Wednesday 15 February 2006 08:21, Peter Bengtsson wrote: class PersistentAnything(PersistentMapping, PersistentList, PersistentDict): AA! This is so wrong! It merges two incompatible APIs: collections and mappings. The non-persistent equivalent to this is: class DoEverything(set, list, dict): ... pass Am I just trying to make it too simple? I think you try far too hard to not understand why the persistent mechanism works as it does. You make your life harder than it has to be. Regards, Stephan -- Stephan Richter CBU Physics Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users -- Jeff Shell ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] What attributes are made persistent
Peter Bengtsson wrote: On 2/16/06, Jeff Shell [EMAIL PROTECTED] wrote: One could, but it's really not worth it. It's just the laws of Python and mutability and immutability :). (It took me years to understand those terms. I kept associating them with 'mutable' in the can be made quiet sense... Eventually my music brain stepped back and I went oh, MUTATE! Ahhh!. Seriously, it took me about four years to understand that :). Wow! I think I love you Jeff. Not only do you reply with non-condencending constructive advise you also confess to [past] weaknesses. good try:-) but that's a totally different ethymology, to mute is to make silent, to tone down (muet in French) mutate c.1374, action of changing, from L. mutationem (nom. mutatio) http://www.etymonline.com/index.php?search=mutate /JM ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] What attributes are made persistent
Peter Bengtsson wrote: self.queue seem is empty each time I restart Zope. That's because dictionaries are not derived from Persistent. Try PersistentDict. D'oh! That's confusing. Isn't there a class that gathers all of these in one. Urm, Peter, the rules of persistence w.r.t. mutable objects have been documented for a _long_ time. There's nothing Zope 3-specific here, and someone who's been using Zope as long as you should know better ;-) Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] What attributes are made persistent
-- Forwarded message -- From: Peter Bengtsson [EMAIL PROTECTED] To: Jeff Shell [EMAIL PROTECTED] Date: Wed, 15 Feb 2006 13:14:58 + Subject: Re: [Zope3-Users] What attributes are made persistent On 2/15/06, Jeff Shell [EMAIL PROTECTED] wrote: On 2/14/06, Peter Bengtsson [EMAIL PROTECTED] wrote: D'oh! That's confusing. Isn't there a class that gathers all of these in one. It seems confusing, you derive from Persistent but only some are accepted. Does that mean that there's PersistentFloat and PersistentTuple too? If not, why *only* dicts? As mentioned above, it applies to *mutable* objects. Instances of Persistent based classes know when they change. Like if you do:: clive.age = 28 On the other hand, if you do:: clive.favoriteNumbers.append(13) 'clive' does not change. 'favoriteNumbers' changes. If favoriteNumbers is a regular Python list, the persistence machinery has no idea that it's changed. It's not being assigned to 'clive', it's already an attribute there and is not being replaced. So if it's not a PersistentList, it gets forgotten. It's not only dicts, it's dicts and lists (PersistentDict and PersistentList). I don't know if there's a PersistentSet. Python offers two sets since 2.3 - a mutable (list-like) one and an immutable (tuple-like) one. I imagine that if you use the mutable set (``sets.Set`` in 2.3, ``set`` in 2.4), you'd run into the same problems. But if you used the immutable set (``sets.ImmutableSet``, ``frozenset`` in 2.4) you wouldn't. So - just use PersistentList and PersistentDict (or look into BTrees for better storage options). For more details, visit the ZODB documentation on ZODB programming, and visit the section on modifying mutable objects: http://www.zope.org/Wikis/ZODB/FrontPage/guide/node3.html I understand the mutation stuff and I always do it like this in zope2 (I'm a complete beginner in the zope3 world eager to learn): def updatesometing(self): #self.numbers['Peter'] = 0779 123 456 numbers = self.numbers numbers['Peter'] = 0779 123 456 self.numbers = numbers But in zope2 land, if I derive from Persistent it magically saves ALL attributes defined in __init__ assuming that I post-write to them correct as shown above. I like this simplicity and was hoping to find it in zope3 land too. I guess I'm just an old dog and if I want to drag with me the zope2 way I can do this:: import persistent.mapping.PersistentMapping import persistent.list.PersistentList import persistent.dict.PersistentDict class PersistentAnything(PersistentMapping, PersistentList, PersistentDict): I'm aware of BTrees and am/will always use it when size of objects becomes uncontrollable. Am I just trying to make it too simple? Have I read one too many books by Steve Krug? -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] What attributes are made persistent
On Wed, Feb 15, 2006 at 01:21:14PM +, Peter Bengtsson wrote: I understand the mutation stuff and I always do it like this in zope2 (I'm a complete beginner in the zope3 world eager to learn): def updatesometing(self): #self.numbers['Peter'] = 0779 123 456 numbers = self.numbers numbers['Peter'] = 0779 123 456 self.numbers = numbers But in zope2 land, if I derive from Persistent it magically saves ALL attributes defined in __init__ assuming that I post-write to them correct as shown above. That hasn't changed in zope 3. This is just standard ZODB behavior. setattr will cause a save on commit regardless of the type of the value. If you read Jeff's reply again carefully, he said as much. PersistentList and PersistentDict are not new, either. They've been used in Zope 2 projects for ages. There are still exactly three ways to get sub-object mutations to persist: 1) Set the dirty bit by hand, e.g.: self.alist.append(1) self._p_changed = 1 2) Re-assign the attribute, e.g.: alist = self.alist alist.append(1) self.alist = alist 3) Using a persistent sub-object, e.g. a PersistentList instance: self.alist.append(1) -- Paul Winkler http://www.slinkp.com ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] What attributes are made persistent
On 2/15/06, Paul Winkler [EMAIL PROTECTED] wrote: On Wed, Feb 15, 2006 at 01:21:14PM +, Peter Bengtsson wrote: I understand the mutation stuff and I always do it like this in zope2 (I'm a complete beginner in the zope3 world eager to learn): def updatesometing(self): #self.numbers['Peter'] = 0779 123 456 numbers = self.numbers numbers['Peter'] = 0779 123 456 self.numbers = numbers But in zope2 land, if I derive from Persistent it magically saves ALL attributes defined in __init__ assuming that I post-write to them correct as shown above. That hasn't changed in zope 3. This is just standard ZODB behavior. setattr will cause a save on commit regardless of the type of the value. If you read Jeff's reply again carefully, he said as much. PersistentList and PersistentDict are not new, either. They've been used in Zope 2 projects for ages. There are still exactly three ways to get sub-object mutations to persist: 1) Set the dirty bit by hand, e.g.: self.alist.append(1) self._p_changed = 1 2) Re-assign the attribute, e.g.: alist = self.alist alist.append(1) self.alist = alist 3) Using a persistent sub-object, e.g. a PersistentList instance: self.alist.append(1) Now I get it! Sorry for being slow and thanks for explaining. So by using PersistentList it just means that you can use: self.alist.append(1) in your code. The attribute, self.alist, is still saved even without PersistentList but it just means you have to be careful when writing to it. Cool! -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] What attributes are made persistent
On 2/15/06, Peter Bengtsson [EMAIL PROTECTED] wrote: D'oh! That's confusing. Isn't there a class that gathers all of these in one. All of who? It seems confusing, you derive from Persistent but only some are accepted. Does that mean that there's PersistentFloat and PersistentTuple too? Because floats and tuples are not mutable. You can't change them, so the don't need to know that they should be persisted when changed. If not, why *only* dicts? There is also PersistentList. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
RE: [Zope3-Users] What attributes are made persistent
Well, one could have a base class along the lines of class PersistSetItemOnAttributes: def __setattr__( self, attr, val ): oldSI = val.__class__.__dict__.get( '__setitem__', None ) if oldSI is not None: def newSI( vself, vattr, vval ): vself._p_changed = True # is this the right member? oldSI( vattr, vval )# oldSI is bound: no vself? val.__class__.__setitem__ = newSI super( PersistSetItemOnAttributes, self ).__setattr__( attr, val ) Of course, this is really just pseudocode. For one thing, need to trap whether 'val' is really a class. For another if you were serious about this, you would want to check if the obj wasn't persistent first, and you might want to do it recursively. And while your are at it, you might want to check on other mutators as well(for instance, check first what sequence interface if any 'val' supports...) ... sounds like too much work, and would be problem prone even so. After all, some things you don't want to be persistent! - Shaun -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Stephan Richter Sent: Wednesday, February 15, 2006 8:43 AM To: zope3-users@zope.org Cc: Florian Lindner Subject: Re: [Zope3-Users] What attributes are made persistent On Wednesday 15 February 2006 08:21, Peter Bengtsson wrote: class PersistentAnything(PersistentMapping, PersistentList, PersistentDict): AA! This is so wrong! It merges two incompatible APIs: collections and mappings. The non-persistent equivalent to this is: class DoEverything(set, list, dict): ... pass Am I just trying to make it too simple? I think you try far too hard to not understand why the persistent mechanism works as it does. You make your life harder than it has to be. Regards, Stephan -- Stephan Richter CBU Physics Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
[Zope3-Users] What attributes are made persistent
Hello,
in a class derived from Persistent, which attributes are stored? All or only
those thar are declared in the interface?
def __init__(self):
self.queue = {}
self.queue seem is empty each time I restart Zope.
Thanks,
Florian
___
Zope3-users mailing list
Zope3-users@zope.org
http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] What attributes are made persistent
On Feb 14, 2006, at 4:52 PM, Florian Lindner wrote:
Hello,
in a class derived from Persistent, which attributes are stored?
All or only
those thar are declared in the interface?
All.
def __init__(self):
self.queue = {}
self.queue seem is empty each time I restart Zope.
Mutable objects, such as sets, lists, and dicts, can't inform their
persistent parents when they change, so changes to them are not
persisted. Use a persistent mapping in the persistent package (or a
btree); or set obj._p_changed=True when the dict changes; or use an
immutable data structure that must be replaced on the object instead.
Gary
___
Zope3-users mailing list
Zope3-users@zope.org
http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] What attributes are made persistent
On 2/14/06, Florian Lindner [EMAIL PROTECTED] wrote:
Hello,
in a class derived from Persistent, which attributes are stored? All or only
those thar are declared in the interface?
All.
def __init__(self):
self.queue = {}
self.queue seem is empty each time I restart Zope.
That's because dictionaries are not derived from Persistent. Try PersistentDict.
--
Lennart Regebro, Nuxeo http://www.nuxeo.com/
CPS Content Management http://www.cps-project.org/
___
Zope3-users mailing list
Zope3-users@zope.org
http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] What attributes are made persistent
On 2/14/06, Lennart Regebro [EMAIL PROTECTED] wrote:
On 2/14/06, Florian Lindner [EMAIL PROTECTED] wrote:
Hello,
in a class derived from Persistent, which attributes are stored? All or only
those thar are declared in the interface?
All.
def __init__(self):
self.queue = {}
self.queue seem is empty each time I restart Zope.
That's because dictionaries are not derived from Persistent. Try
PersistentDict.
D'oh! That's confusing. Isn't there a class that gathers all of these in one.
It seems confusing, you derive from Persistent but only some are accepted.
Does that mean that there's PersistentFloat and PersistentTuple too?
If not, why *only* dicts?
--
Lennart Regebro, Nuxeo http://www.nuxeo.com/
CPS Content Management http://www.cps-project.org/
___
Zope3-users mailing list
Zope3-users@zope.org
http://mail.zope.org/mailman/listinfo/zope3-users
--
Peter Bengtsson,
work www.fry-it.com
home www.peterbe.com
hobby www.issuetrackerproduct.com
___
Zope3-users mailing list
Zope3-users@zope.org
http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] What attributes are made persistent
Peter Bengtsson wrote:
def __init__(self):
self.queue = {}
self.queue seem is empty each time I restart Zope.
That's because dictionaries are not derived from Persistent. Try PersistentDict.
D'oh! That's confusing. Isn't there a class that gathers all of these in one.
It seems confusing, you derive from Persistent but only some are accepted.
Does that mean that there's PersistentFloat and PersistentTuple too?
If not, why *only* dicts?
No, this issue applies to mutable attributes..
self.a = 1.1
self.b = (1,2,3)
self.b = (4,5,6)
These will persist as is, because you're rebinding the attribute
(reference) each time.
However, for ...
self.queue[1] = 2
This doesn't rebind self.queue.
So another way you could 'persist' this change is ..
self.queue[1] = 1
self.queue = self.queue
___
Zope3-users mailing list
Zope3-users@zope.org
http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] What attributes are made persistent
On 2/14/06, Peter Bengtsson [EMAIL PROTECTED] wrote: D'oh! That's confusing. Isn't there a class that gathers all of these in one. It seems confusing, you derive from Persistent but only some are accepted. Does that mean that there's PersistentFloat and PersistentTuple too? If not, why *only* dicts? As mentioned above, it applies to *mutable* objects. Objects whose value can change are said to be mutable; objects whose value is unchangeable once they are created are called immutable. (The value of an immutable container object that contains a reference to a mutable object can change when the latter's value is changed; however the container is still considered immutable, because the collection of objects it contains cannot be changed. So, immutability is not strictly the same as having an unchangeable value, it is more subtle.) An object's mutability is determined by its type; for instance, numbers, strings and tuples are immutable, while dictionaries and lists are mutable. http://docs.python.org/ref/objects.html Instances of Persistent based classes know when they change. Like if you do:: clive.age = 28 Which is effectively ``setattr(clive, 'age', 28)``. As a persistent object, clive notices that an attribute is being set (even if it's replacing an existing value), and flags that 'clive' has changes that need to be saved. So again - here, it's the 'clive' object that's being updated in this case. The fact that the value is an integer doesn't matter. It's not changing. 28 is 28. But 'clive' is changing by having a new/updated 'age' attribute set. On the other hand, if you do:: clive.favoriteNumbers.append(13) 'clive' does not change. 'favoriteNumbers' changes. If favoriteNumbers is a regular Python list, the persistence machinery has no idea that it's changed. It's not being assigned to 'clive', it's already an attribute there and is not being replaced. So if it's not a PersistentList, it gets forgotten. If favoriteNumbers was a tuple, you couldn't append to it. You could do an addition, but in a form like this:: clive.favoriteNumbers = clive.favoriteNumbers + (13,) # Alternately clive.favoriteNumbers += (13,) This is doing assignment. 'clive' is getting a new attribute set, and as a Persistent object 'clive' can be set as needing its changes saved. It's not only dicts, it's dicts and lists (PersistentDict and PersistentList). I don't know if there's a PersistentSet. Python offers two sets since 2.3 - a mutable (list-like) one and an immutable (tuple-like) one. I imagine that if you use the mutable set (``sets.Set`` in 2.3, ``set`` in 2.4), you'd run into the same problems. But if you used the immutable set (``sets.ImmutableSet``, ``frozenset`` in 2.4) you wouldn't. Note that this issue affects not just ZODB persistence. If you used even the simple 'shelve' module that's in Python, you have the same issue unless you use a 'writeback' option: If the writeback parameter is True, the object will hold a cache of all entries accessed and write them back to the dict at sync and close times. This allows natural operations on mutable entries, but can consume much more memory and make sync and close take a long time. So as a way of making working with default mutable objects feel natural, this option makes shelve read and write its whole database, which would obviously be very expensive for the ZODB. So - just use PersistentList and PersistentDict (or look into BTrees for better storage options). For more details, visit the ZODB documentation on ZODB programming, and visit the section on modifying mutable objects: http://www.zope.org/Wikis/ZODB/FrontPage/guide/node3.html ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] What attributes are made persistent
Oh! And reference to related Persistence modules (persistent mappings, lists, BTrees): http://www.zope.org/Wikis/ZODB/FrontPage/guide/node6.html On 2/14/06, Jeff Shell [EMAIL PROTECTED] wrote: On 2/14/06, Peter Bengtsson [EMAIL PROTECTED] wrote: D'oh! That's confusing. Isn't there a class that gathers all of these in one. It seems confusing, you derive from Persistent but only some are accepted. Does that mean that there's PersistentFloat and PersistentTuple too? If not, why *only* dicts? As mentioned above, it applies to *mutable* objects. Objects whose value can change are said to be mutable; objects whose value is unchangeable once they are created are called immutable. (The value of an immutable container object that contains a reference to a mutable object can change when the latter's value is changed; however the container is still considered immutable, because the collection of objects it contains cannot be changed. So, immutability is not strictly the same as having an unchangeable value, it is more subtle.) An object's mutability is determined by its type; for instance, numbers, strings and tuples are immutable, while dictionaries and lists are mutable. http://docs.python.org/ref/objects.html Instances of Persistent based classes know when they change. Like if you do:: clive.age = 28 Which is effectively ``setattr(clive, 'age', 28)``. As a persistent object, clive notices that an attribute is being set (even if it's replacing an existing value), and flags that 'clive' has changes that need to be saved. So again - here, it's the 'clive' object that's being updated in this case. The fact that the value is an integer doesn't matter. It's not changing. 28 is 28. But 'clive' is changing by having a new/updated 'age' attribute set. On the other hand, if you do:: clive.favoriteNumbers.append(13) 'clive' does not change. 'favoriteNumbers' changes. If favoriteNumbers is a regular Python list, the persistence machinery has no idea that it's changed. It's not being assigned to 'clive', it's already an attribute there and is not being replaced. So if it's not a PersistentList, it gets forgotten. If favoriteNumbers was a tuple, you couldn't append to it. You could do an addition, but in a form like this:: clive.favoriteNumbers = clive.favoriteNumbers + (13,) # Alternately clive.favoriteNumbers += (13,) This is doing assignment. 'clive' is getting a new attribute set, and as a Persistent object 'clive' can be set as needing its changes saved. It's not only dicts, it's dicts and lists (PersistentDict and PersistentList). I don't know if there's a PersistentSet. Python offers two sets since 2.3 - a mutable (list-like) one and an immutable (tuple-like) one. I imagine that if you use the mutable set (``sets.Set`` in 2.3, ``set`` in 2.4), you'd run into the same problems. But if you used the immutable set (``sets.ImmutableSet``, ``frozenset`` in 2.4) you wouldn't. Note that this issue affects not just ZODB persistence. If you used even the simple 'shelve' module that's in Python, you have the same issue unless you use a 'writeback' option: If the writeback parameter is True, the object will hold a cache of all entries accessed and write them back to the dict at sync and close times. This allows natural operations on mutable entries, but can consume much more memory and make sync and close take a long time. So as a way of making working with default mutable objects feel natural, this option makes shelve read and write its whole database, which would obviously be very expensive for the ZODB. So - just use PersistentList and PersistentDict (or look into BTrees for better storage options). For more details, visit the ZODB documentation on ZODB programming, and visit the section on modifying mutable objects: http://www.zope.org/Wikis/ZODB/FrontPage/guide/node3.html -- Jeff Shell ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
