Re: [Zope-dev] Help: __getstate__ overriding
"Tim Peters" <[EMAIL PROTECTED]> writes: > That adds something to the BTree that is bound to an attribute of articleDb, > but still doesn't set _p_changed on articleDb (mutating the BTree does not > mutate the object containing the BTree). So far as the *database* is > concerned, there's still no path from the root object to the BTree your > __setstate__() created, so the BTree never gets committed. So the BTree is not "registered" with the database until _p_changed and a commit? After that first commit, the BTree is "registered" as reachable and then when just the BTree is _p_changed I get my changes saved. Right? > > Calling __setstate__ as a side effect of a persistent load does not, by > itself, support making changes that persist. It apparently wasn't designed > to, either (see the other msgs in this thread, particularly from ChrisM). I can understand the reasoning behind it, in the way that you don't want nothing to change on disk when you logically haven't called a mutating method on your object, just loaded it. But I do think it is badly broken to not save the changes when I have made additional changes to the state of the object after load time. Why not just take the performance hit and be done with it? What do other object databases do with schema migration issues, f.ex. GemStone? Now well, I believe I understand why this happens and that is a very good thing! Thanks to all who have helped me in gaining a better understanding of ZODB. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Help: __getstate__ overriding
"Tim Peters" <[EMAIL PROTECTED]> writes: > [Syver Enstad, wants to switch an attribute of a Persitent subclass > From PersistentList to an IOBTree] > > [Tim, guessing] > > Quick guess (untested, untried, ... > ... > > Perhaps shuffling the code around would work, a la: > > > > Persistent.__setstate__(self, state) > > articleList = state.get('_articleList') > > if articleList is not None: > > # make the BTree in local oidsToArticles as above, then > > del self._articlelist > > self._oidsToArticles = oidsToArticles > > > > The thinking there is that then you've truly modified self, after self has > > been activated. > > Nope, sorry, not a chance. You have truly modified self then, but > __setstate__ is called by magic as part of activation (unghostifying), and > the activation machinery resets self._p_changed after it calls __setstate__. > So same result as your code: the in-memory version of self has the BTree, > but commit() doesn't change anything about self in the database (again > because self._p_changed is false -- and will be no matter what you try to do > in __setstate__()). I was aware that __setstate__ doesn't allow me to commit my changes after only loading the object into memory (__setstate__ is called). I may have explained myself unclearly (not a native english speaker/writer), the problem is that it won't save my changes when I add more items to the _oidsToArticles BTree long after __setstate__ time. Example code: # articleDb is an ArticleDb and has the __setstate__ method articleDb = connection.root()['articledb'] for each in articleDb.articles() # loop through all article print each computer = Computer(1030) articleDb.addArticle(computer) connection.commit() # nothing is saved Everything works smoothly until connection.commit(). The __setstate__ method correctly converts to using the BTree and the addArticle method adds a new computer to ArticleDb. If I instead create a new ArticleDb instance with the BTree instead of using __setstate__ the new computer is saved as it should. In none of the cases is articleDb._p_changed == True, but when the ArticleDb is created "cleanly" instead of upgraded, commit works anyway. Since this works if the object that holds _oidsToArticles is created from scratch, it seems that ZODB isn't properly aware of the _oidsToArticles attribute since it was created in __setstate__ instead of in __init__. I would really like to know exactly what happens here as the reason I am utilizing an object database is to be able to refactor the database with a minimum of fuss, and I would expect __setstate__ to allow me to make additional attributes in my instances. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Help: __getstate__ overriding
I have a Persistent derived class where I want to upgrade from using a PersistentList to a BTrees.IOBTree.IOBTree. _articleList is the old Persistent list _oidsToArticles is the new IOBTree. def __setstate__(self, state): articleList = state.get('_articleList') if articleList: oidsToArticles = self.makeBTree() for each in articleList: oidsToArticles[each.oid()] = each del state['_articleList'] state['_oidsToArticles'] = oidsToArticles Persistent.__setstate__(self, state) This seems to work fine, and I am adding more objects to _oidsToArticles. The problem is that when I commit the transaction, nothing is saved. I have tested the same code without __setstate__ and then it saves just fine. What hoops does one have to jump through to upgrade the state of a ZODB object? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] performance tuning of ZODB
Leonardo Rochael Almeida <[EMAIL PROTECTED]> writes: > Hi Syver, > > Please add this issue to the Collector, including the test (prefereably > without the twisted bits) > Eh, what is the Collector? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: When should one call Connection.sync?
Dieter Maurer <[EMAIL PROTECTED]> writes: > Syver Enstad wrote at 2004-4-21 18:03 +0200: > >I have done some experiments with this scheme and I find that > >everything gets unloaded when do a connection.close() or > >connection.sync() so that performance takes quite a hit. > > Maybe, the connection cache is too small. > > Usually, an incremental cache garbage collection is performed > in "connection.close()" and at transaction boundaries > ("sync" causes an implicit "transaction.abort()" which means, > it marks a transaction boundary). The cache garbage collection > tries to flush as many objects from the cache as are necessary > to reach the target cache size. Yes that was the problem, I found out about this after some experiments and by upping the cache_size to at least 20 000 I manage to keep the instances in memory even when I do a sync or close. Thank you anyway. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] performance tuning of ZODB
Toby Dickenson <[EMAIL PROTECTED]> writes: > On Thursday 22 April 2004 11:43, Syver Enstad wrote: > > > cache_deactivate_after sounds interesting. Since I am running ZODB in > > a web server I don't want the data to timeout and disappear from > > memory since every object I have should be loaded at all times. > > Thats why ZODB ignores that parameter now ;-) Good :-) I have a strange case here with ReadConflictErrors. I don't know if this is covered already but anyway. I am using ZODB 3.2 so this might be fixed in 3.3 for all I know. Just replace the twisted stuff with the standard lib unittest module to have it work on a computer without twisted installed. import pdb import sys import time import threading import os from ZODB import DB from ZODB.PersistentList import PersistentList from ZODB.FileStorage import FileStorage from ZODB.POSException import ReadConflictError from Persistence import Persistent from twisted.trial import unittest class Computer(Persistent): def __init__(self, oidInteger): self._oid = oidInteger self._articleStatus = None self._endUserPrice = None self._location = None def setArticleStatus(self, anObject): self._articleStatus = anObject def setEndUserPrice(self, anObject): self._endUserPrice = anObject def setLocation(self, anObject): self._location = anObject class MockArticleDb(Persistent): def __init__(self): self._articles = PersistentList() def addArticle(self, anArticle): self._articles.append(anArticle) def articles(self): return self._articles class TestReadConflictError(unittest.TestCase): def setUp(self): try: os.unlink('test.fs') except OSError, err: pass storage = FileStorage( 'test.fs', create=True) db = DB(storage, cache_size=10) connection = db.open() connection.setLocalTransaction() connection.root()['articledb'] = MockArticleDb() computer = Computer(1) computer.setArticleStatus('For sale') computer.setEndUserPrice(10050) computer.setLocation('H0101') connection.root()['articledb'].addArticle(computer) connection.getTransaction().commit() connection.close() db.close() storage.close() storage = FileStorage( 'test.fs', create=False) self._db = DB(storage, cache_size=10) def writeStuff(self, connection): begin = time.time() connection.sync() articleDb = connection.root()['articledb'] eachBegin = time.time() computer = Computer(1) computer.setArticleStatus('For sale') computer.setEndUserPrice(10050) computer.setLocation('H0101') articleDb.addArticle(computer) connection.getTransaction().commit() print 'write list length', len(articleDb.articles()) def doRead(self, oddRead, connection): tryCount = 1 articleDb = connection.root()['articledb'] try: for article in articleDb.articles(): pass except ReadConflictError, err: if True: connection.sync() else: # this also works connection.close() connection = self._db.open() connection.setLocalTransaction() for article in articleDb.articles(): pass tryCount += 1 print 'read list length', len(articleDb.articles()) if oddRead: # on first, third, fifth read we get a ReadConflictError self.assertEquals( 2, tryCount) else: # but on second, fourth and so forth we don't self.assertEquals( 1, tryCount) def testProvokeReadConflictError(self): conn2 = self._db.open() conn2.setLocalTransaction() for each in range(1, 11): # 1000 works also but takes a bit of time begin = time.time() connection = self._db.open() connection.setLocalTransaction() self.writeStuff(conn2) oddRead = each % 2 != 0 self.doRead(oddRead, connection) connection.close() if __name__ == '__main__': from twisted.scripts.trial import run import sys sys.argv.append(sys.argv[0]) run() ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] performance tuning of ZODB
Toby Dickenson <[EMAIL PROTECTED]> writes: > On Thursday 22 April 2004 10:42, Syver Enstad wrote: > > I've used the cache_size paramter to the constructor of the DB to good > > effect. Are there any further gotchas for ensuring that the ZODB stays > > in memory as much as possible? > > Memory usage scales proportional to the number of threads. Reducing the > number of worker threads lets you increase the per-thread cache size. I don't use connection per thread semantics but I think I understand what you mean. I should pass a lower number than the default to the pool_size keyword argument for the DB constructor? What about the other arguments to the DB constructor? cache_deactivate_after sounds interesting. Since I am running ZODB in a web server I don't want the data to timeout and disappear from memory since every object I have should be loaded at all times. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] performance tuning of ZODB
I've used the cache_size paramter to the constructor of the DB to good effect. Are there any further gotchas for ensuring that the ZODB stays in memory as much as possible? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: When should one call Connection.sync?
Syver Enstad <[EMAIL PROTECTED]> writes: > I am using ZODB 3.2 in a twisted based web application. I have read > that I need to call sync to keep the connection up to date. When > exactly should I call sync? Are there any drawbacks with calling it > immediately after getting a connection, like this: > > # for each http request. > connection = db.open() # (a DB instance) > connection.setLocalTransaction() > connection.sync() > > # start using the ZODB here. > > # if something needs to be committed > connection.getTransaction().commit() I have done some experiments with this scheme and I find that everything gets unloaded when do a connection.close() or connection.sync() so that performance takes quite a hit. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] When should one call Connection.sync?
I am using ZODB 3.2 in a twisted based web application. I have read that I need to call sync to keep the connection up to date. When exactly should I call sync? Are there any drawbacks with calling it immediately after getting a connection, like this: # for each http request. connection = db.open() # (a DB instance) connection.setLocalTransaction() connection.sync() # start using the ZODB here. # if something needs to be committed connection.getTransaction().commit() ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] class methods in Persistent objects
Is it possible to have class methods on ZODB Persistent objects? I am using 3.2 so that means I am using ExtensionClass. I guess that moving to 3.3 would mean being able to use class methods? What is the expected time that the 3.3 release will be ready for production use? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Unique ID generation with ZODB.
Andreas Jung <[EMAIL PROTECTED]> writes: > The simplest solution is to keep the counter as file in the > filesystem. Access to the file > must be synchronized through a lock object (-> Python docs). Just a > few lines of Python > code. It has to work in windows too, so it's a bit more involved but should be manageable with win32all. I don't particularely like that I have another piece of data to backup though. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Unique ID generation with ZODB.
How does one implement a unique id generator in ZODB. The id's should be strictyl ascending and be a an int. Would opening a temporary connection increment a persistent counter and commit do the trick? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZODB with twisted web.
I am checking out how to use ZODB with twisted web. I thought that I would have the DB instance globally accesible and call the open method to get a connection on each request. I thought that if I use connection.getTransaction().commit() before sending the response I will commit on the connection for that request. Is this necesarily true when using twisted web because it is single threaded and it seems that ZODB finds the correct transaction by checking the thread. I am a bit confused here and would rather like to avoid having one request commit changes for another request. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: [ZODB-Dev] Schema migration, rename class
Jeremy Hylton <[EMAIL PROTECTED]> writes: > On Wed, 2004-03-31 at 09:19, Syver Enstad wrote: > > I am looking for good resources on schema migration with ZODB. > > > > http://www-106.ibm.com/developerworks/linux/library/l-pypers.html > > > > Shows some of the techniques (the article is about Pickling) The > > article describes how to rename a class by changing the self.__class__ > > to the renamed class in __setstate__ but this doesn't seem to work. > > (I haven't followed the link yet, but ...) You can't change the class > of a persistent object, although it may seem like you can because you > can assign to __class__. I am not really interested in changing the class. I am interested in renaming it. Are there any resources on schema evolution/migration with ZODB, I have googled quite a bit but so far haven't come up with anything other than the link above which only seems to work with pickles. > The problem is that persistent references > contain the class and oid of the object. So you can change __class__, > but the next time the object is loaded it will get whatever class its > referrer thinks it has. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )