Re: [Zope-dev] Help: __getstate__ overriding
Tim Peters [EMAIL PROTECTED] writes: That adds something to the BTree that is bound to an attribute of articleDb, but still doesn't set _p_changed on articleDb (mutating the BTree does not mutate the object containing the BTree). So far as the *database* is concerned, there's still no path from the root object to the BTree your __setstate__() created, so the BTree never gets committed. So the BTree is not registered with the database until _p_changed and a commit? After that first commit, the BTree is registered as reachable and then when just the BTree is _p_changed I get my changes saved. Right? Calling __setstate__ as a side effect of a persistent load does not, by itself, support making changes that persist. It apparently wasn't designed to, either (see the other msgs in this thread, particularly from ChrisM). I can understand the reasoning behind it, in the way that you don't want nothing to change on disk when you logically haven't called a mutating method on your object, just loaded it. But I do think it is badly broken to not save the changes when I have made additional changes to the state of the object after load time. Why not just take the performance hit and be done with it? What do other object databases do with schema migration issues, f.ex. GemStone? Now well, I believe I understand why this happens and that is a very good thing! Thanks to all who have helped me in gaining a better understanding of ZODB. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Help: __getstate__ overriding
Tim Peters wrote at 2004-5-31 20:49 -0400: ... Perhaps we could say that if it returns True, then the object should be considered to be in the changed state rather than in the uptodate state. +1 -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Help: __getstate__ overriding
Tim Peters [EMAIL PROTECTED] writes:
[Syver Enstad, wants to switch an attribute of a Persitent subclass
From PersistentList to an IOBTree]
[Tim, guessing]
Quick guess (untested, untried, ...
...
Perhaps shuffling the code around would work, a la:
Persistent.__setstate__(self, state)
articleList = state.get('_articleList')
if articleList is not None:
# make the BTree in local oidsToArticles as above, then
del self._articlelist
self._oidsToArticles = oidsToArticles
The thinking there is that then you've truly modified self, after self has
been activated.
Nope, sorry, not a chance. You have truly modified self then, but
__setstate__ is called by magic as part of activation (unghostifying), and
the activation machinery resets self._p_changed after it calls __setstate__.
So same result as your code: the in-memory version of self has the BTree,
but commit() doesn't change anything about self in the database (again
because self._p_changed is false -- and will be no matter what you try to do
in __setstate__()).
I was aware that __setstate__ doesn't allow me to commit my changes
after only loading the object into memory (__setstate__ is called). I
may have explained myself unclearly (not a native english
speaker/writer), the problem is that it won't save my changes when I
add more items to the _oidsToArticles BTree long after
__setstate__ time.
Example code:
# articleDb is an ArticleDb and has the __setstate__ method
articleDb = connection.root()['articledb']
for each in articleDb.articles() # loop through all article
print each
computer = Computer(1030)
articleDb.addArticle(computer)
connection.commit() # nothing is saved
Everything works smoothly until connection.commit(). The __setstate__
method correctly converts to using the BTree and the addArticle method
adds a new computer to ArticleDb.
If I instead create a new ArticleDb instance with the BTree instead of
using __setstate__ the new computer is saved as it should. In none of
the cases is articleDb._p_changed == True, but when the ArticleDb is
created cleanly instead of upgraded, commit works anyway.
Since this works if the object that holds _oidsToArticles is created
from scratch, it seems that ZODB isn't properly aware of the
_oidsToArticles attribute since it was created in __setstate__ instead
of in __init__. I would really like to know exactly what happens here
as the reason I am utilizing an object database is to be able to
refactor the database with a minimum of fuss, and I would expect
__setstate__ to allow me to make additional attributes in my
instances.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Help: __getstate__ overriding
[Syver Enstad] I was aware that __setstate__ doesn't allow me to commit my changes after only loading the object into memory (__setstate__ is called). I may have explained myself unclearly (not a native english speaker/writer), I don't think that matters much: English instead of code is always ambiguous, even for native writers. the problem is that it won't save my changes when I add more items to the _oidsToArticles BTree long after __setstate__ time. Right. I tried to explain that in my first reply. It's expected. You have to get _p_changed set on your *object*. It doesn't matter how much you mutate new objects attached *to* the object you loaded, you have to get _p_changed set to 1 on the original object. Example code: # articleDb is an ArticleDb and has the __setstate__ method articleDb = connection.root()['articledb'] At this point articleDb._p_changed is not 1, and nothing you do later changes this fact. for each in articleDb.articles() # loop through all article print each It's obvious that this doesn't set articleDb._p_changed, right? computer = Computer(1030) articleDb.addArticle(computer) That adds something to the BTree that is bound to an attribute of articleDb, but still doesn't set _p_changed on articleDb (mutating the BTree does not mutate the object containing the BTree). So far as the *database* is concerned, there's still no path from the root object to the BTree your __setstate__() created, so the BTree never gets committed. If you do articleDb._p_changed = 1 at this point too, then your changes will get stored. connection.commit() # nothing is saved Right. As far as the persistence machinery is concerned, articleDb itself never changed, so there was no reason to store it. Since it didn't get stored, the BTree you attached to it in __setstate__ doesn't get stored either. Everything works smoothly until connection.commit(). The __setstate__ method correctly converts to using the BTree and the addArticle method adds a new computer to ArticleDb. The problem is that, while adding the BTree to articleDb did change (did set _p_changed to 1) articleDb, because that happened as a side effect of loading the object, the unghostification machinery clears _p_changed after __setstate__ returns. Then nothing you did after that set articleDb._p_changed to a true value. If I instead create a new ArticleDb instance with the BTree instead of using __setstate__ the new computer is saved as it should. Yes. If you had called __setstate__ *directly*, it would also work (contradicting the current minimal docs, but so it goes). The problem is that unghostifying by magic clears _p_changed. In none of the cases is articleDb._p_changed == True, When you create an object directly, it's not associated with a Connection (a jar), and some of the persistence machinery is sidestepped then. New objects only get into the database if you attach them to an object that's already in the database, and the latter object is marked changed. Think about that wink, and the purpose of the root() object will become clearer. but when the ArticleDb is created cleanly instead of upgraded, commit works anyway. Sorry, I'm not clear on what that means. In any case, it doesn't really matter how an object gets created, what matters is whether the persistence machinery sees a path to it from a changed object already in the database. Since this works if the object that holds _oidsToArticles is created from scratch, it seems that ZODB isn't properly aware of the _oidsToArticles attribute since it was created in __setstate__ instead of in __init__. Both ways set _p_changed *at the time* oidsToArticles is bound to the new BTree. The difference is that uhghostification deliberately clears _p_changed after calling __setstate__. I would really like to know exactly what happens here as the reason I am utilizing an object database is to be able to refactor the database with a minimum of fuss, and I would expect __setstate__ to allow me to make additional attributes in my instances. Calling __setstate__ as a side effect of a persistent load does not, by itself, support making changes that persist. It apparently wasn't designed to, either (see the other msgs in this thread, particularly from ChrisM). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Help: __getstate__ overriding
Tim Peters wrote at 2004-5-30 14:49 -0400: [Dieter Maurer] I think, this is a ZODB buglet: It should set _p_changed = 0 before it calls __setstate__ and not afterwards... I don't know; Jim (or Jeremy) may know the reasoning here, but I don't. Activation currently sets the state to changed *before* calling __setstate__ too. A comment says this is to prevent recursively calling _PyPersist_Load, and that makes sense to me, else unbounded recursion could occur. I see... ... On ZODB head, Jeremy also documented that __setstate__ can't affect the persistent state (in persistent/interfaces.py's IPersistent). Whether this is deliberate design, or a consequence of the current implementation, I can't say. A potential problem: it *can* change the persistent state but it does not do it reliably. It does precisely when the object is later modified again: in this case modifications made in __setstate__ become persistent, otherwise, they do not. -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Help: __getstate__ overriding
[Tim Peters] ... On ZODB head, Jeremy also documented that __setstate__ can't affect the persistent state (in persistent/interfaces.py's IPersistent). Whether this is deliberate design, or a consequence of the current implementation, I can't say. [Dieter Maurer] A potential problem: it *can* change the persistent state but it does not do it reliably. It does precisely when the object is later modified again: in this case modifications made in __setstate__ become persistent, otherwise, they do not. I agree, but the docs are cryptic (because so brief) enough that I can read them as being correct or incorrect: def __setstate__(state): Set the object state data Note that this does not affect the object's persistent state. There are enough ambiguities there to fill a whole thread wink. It is true that a call to __setstate__() alone doesn't affect the object's state in the DB. It needs more words. OTOH, I'd rather dream up a way to make object changes made in __setstate__() persistent, if desired. For example, __setstate__'s return value is ignored now. Perhaps we could say that if it returns True, then the object should be considered to be in the changed state rather than in the uptodate state. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Help: __getstate__ overriding
Tim Peters wrote at 2004-5-28 14:51 -0400: ... This appears to be one of those severely underdocumented minefields. The best older thread I found is here: http://mail.zope.org/pipermail/zodb-dev/2002-March/002442.html but it doesn't actually spell out something that works in the way you want. I think, this is a ZODB buglet: It should set _p_changed = 0 before it calls __setstate__ and not afterwards... -- Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Help: __getstate__ overriding
[Dieter Maurer] I think, this is a ZODB buglet: It should set _p_changed = 0 before it calls __setstate__ and not afterwards... I don't know; Jim (or Jeremy) may know the reasoning here, but I don't. Activation currently sets the state to changed *before* calling __setstate__ too. A comment says this is to prevent recursively calling _PyPersist_Load, and that makes sense to me, else unbounded recursion could occur. Given this abuse of _p_changed, and that the purpose of activating a ghost is to transition it to the up-to-date state, making mutations to the persistent state inside __setstate__ seems hard to accommodate. On ZODB head, Jeremy also documented that __setstate__ can't affect the persistent state (in persistent/interfaces.py's IPersistent). Whether this is deliberate design, or a consequence of the current implementation, I can't say. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Help: __getstate__ overriding
I have a Persistent derived class where I want to upgrade from using a
PersistentList to a BTrees.IOBTree.IOBTree.
_articleList is the old Persistent list
_oidsToArticles is the new IOBTree.
def __setstate__(self, state):
articleList = state.get('_articleList')
if articleList:
oidsToArticles = self.makeBTree()
for each in articleList:
oidsToArticles[each.oid()] = each
del state['_articleList']
state['_oidsToArticles'] = oidsToArticles
Persistent.__setstate__(self, state)
This seems to work fine, and I am adding more objects to
_oidsToArticles. The problem is that when I commit the transaction,
nothing is saved. I have tested the same code without __setstate__ and
then it saves just fine. What hoops does one have to jump through to
upgrade the state of a ZODB object?
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Help: __getstate__ overriding
[Syver Enstad]
I have a Persistent derived class where I want to upgrade from using a
PersistentList to a BTrees.IOBTree.IOBTree.
_articleList is the old Persistent list
_oidsToArticles is the new IOBTree.
def __setstate__(self, state):
articleList = state.get('_articleList')
if articleList:
oidsToArticles = self.makeBTree()
for each in articleList:
oidsToArticles[each.oid()] = each
del state['_articleList']
state['_oidsToArticles'] = oidsToArticles
Persistent.__setstate__(self, state)
This seems to work fine, and I am adding more objects to
_oidsToArticles. The problem is that when I commit the transaction,
nothing is saved.
Quick guess (untested, untried, just from eyeballing this snippet quickly):
nothing is marking self itself as being changed. The only things getting
changed are the throw-away in-memory 'state' dict, the throw-away in-memory
self.__dict__, and a throw-away in-memory BTree. The primary purpose of
p.__setstate__ is to activate a ghost, after which point p is in the
up-to-date state (i.e., not changed -- calling Persistent.__setstate__(self,
...) isn't going to mark self as changed, and nothing in the code above
modifies self itself). If self doesn't look changed, its state in the DB
won't change during commit(), and the DB will still reference your old
PersistentList.
Perhaps shuffling the code around would work, a la:
Persistent.__setstate__(self, state)
articleList = state.get('_articleList')
if articleList is not None:
# make the BTree in local oidsToArticles as above, then
del self._articlelist
self._oidsToArticles = oidsToArticles
The thinking there is that then you've truly modified self, after self has
been activated.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Help: __getstate__ overriding
[Syver Enstad, wants to switch an attribute of a Persitent subclass
From PersistentList to an IOBTree]
[Tim, guessing]
Quick guess (untested, untried, ...
...
Perhaps shuffling the code around would work, a la:
Persistent.__setstate__(self, state)
articleList = state.get('_articleList')
if articleList is not None:
# make the BTree in local oidsToArticles as above, then
del self._articlelist
self._oidsToArticles = oidsToArticles
The thinking there is that then you've truly modified self, after self has
been activated.
Nope, sorry, not a chance. You have truly modified self then, but
__setstate__ is called by magic as part of activation (unghostifying), and
the activation machinery resets self._p_changed after it calls __setstate__.
So same result as your code: the in-memory version of self has the BTree,
but commit() doesn't change anything about self in the database (again
because self._p_changed is false -- and will be no matter what you try to do
in __setstate__()).
This appears to be one of those severely underdocumented minefields. The
best older thread I found is here:
http://mail.zope.org/pipermail/zodb-dev/2002-March/002442.html
but it doesn't actually spell out something that works in the way you
want.
One possibility is to use any of these __setstate__ implementations
temporarily, in a one-shot database conversion script: load every object of
your subclass's type, and for each one obj, after loading it do
obj._p_changed = True
get_transaction().commit()
Note again that setting _p_changed *inside* __setstate__ won't do any good.
Note too that Jim Fulton has a recent proposal for Zope3 in this area:
http://tinyurl.com/ypkhk
[Zope URLs are generally so long my mailer refuses to keep them on one line]
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Help: __getstate__ overriding
Syver Enstad wrote at 2004-5-28 12:41 +0200:
I have a Persistent derived class where I want to upgrade from using a
PersistentList to a BTrees.IOBTree.IOBTree.
_articleList is the old Persistent list
_oidsToArticles is the new IOBTree.
def __setstate__(self, state):
articleList = state.get('_articleList')
if articleList:
oidsToArticles = self.makeBTree()
for each in articleList:
oidsToArticles[each.oid()] = each
del state['_articleList']
state['_oidsToArticles'] = oidsToArticles
Persistent.__setstate__(self, state)
This seems to work fine, and I am adding more objects to
_oidsToArticles. The problem is that when I commit the transaction,
nothing is saved. I have tested the same code without __setstate__ and
then it saves just fine. What hoops does one have to jump through to
upgrade the state of a ZODB object?
This does not work as changes made in __setstate__ are not
explicitly persisted. Whether or not they become persistent
depends on whether the object is changed again outside __setstate__.
Use an explicit conversion script instead...
--
Dieter
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
