Re: [Zope-dev] Help: __getstate__ overriding
Tim Peters [EMAIL PROTECTED] writes: That adds something to the BTree that is bound to an attribute of articleDb, but still doesn't set _p_changed on articleDb (mutating the BTree does not mutate the object containing the BTree). So far as the *database* is concerned, there's still no path from the root object to the BTree your __setstate__() created, so the BTree never gets committed. So the BTree is not registered with the database until _p_changed and a commit? After that first commit, the BTree is registered as reachable and then when just the BTree is _p_changed I get my changes saved. Right? Calling __setstate__ as a side effect of a persistent load does not, by itself, support making changes that persist. It apparently wasn't designed to, either (see the other msgs in this thread, particularly from ChrisM). I can understand the reasoning behind it, in the way that you don't want nothing to change on disk when you logically haven't called a mutating method on your object, just loaded it. But I do think it is badly broken to not save the changes when I have made additional changes to the state of the object after load time. Why not just take the performance hit and be done with it? What do other object databases do with schema migration issues, f.ex. GemStone? Now well, I believe I understand why this happens and that is a very good thing! Thanks to all who have helped me in gaining a better understanding of ZODB. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Help: __getstate__ overriding
Tim Peters [EMAIL PROTECTED] writes:
[Syver Enstad, wants to switch an attribute of a Persitent subclass
From PersistentList to an IOBTree]
[Tim, guessing]
Quick guess (untested, untried, ...
...
Perhaps shuffling the code around would work, a la:
Persistent.__setstate__(self, state)
articleList = state.get('_articleList')
if articleList is not None:
# make the BTree in local oidsToArticles as above, then
del self._articlelist
self._oidsToArticles = oidsToArticles
The thinking there is that then you've truly modified self, after self has
been activated.
Nope, sorry, not a chance. You have truly modified self then, but
__setstate__ is called by magic as part of activation (unghostifying), and
the activation machinery resets self._p_changed after it calls __setstate__.
So same result as your code: the in-memory version of self has the BTree,
but commit() doesn't change anything about self in the database (again
because self._p_changed is false -- and will be no matter what you try to do
in __setstate__()).
I was aware that __setstate__ doesn't allow me to commit my changes
after only loading the object into memory (__setstate__ is called). I
may have explained myself unclearly (not a native english
speaker/writer), the problem is that it won't save my changes when I
add more items to the _oidsToArticles BTree long after
__setstate__ time.
Example code:
# articleDb is an ArticleDb and has the __setstate__ method
articleDb = connection.root()['articledb']
for each in articleDb.articles() # loop through all article
print each
computer = Computer(1030)
articleDb.addArticle(computer)
connection.commit() # nothing is saved
Everything works smoothly until connection.commit(). The __setstate__
method correctly converts to using the BTree and the addArticle method
adds a new computer to ArticleDb.
If I instead create a new ArticleDb instance with the BTree instead of
using __setstate__ the new computer is saved as it should. In none of
the cases is articleDb._p_changed == True, but when the ArticleDb is
created cleanly instead of upgraded, commit works anyway.
Since this works if the object that holds _oidsToArticles is created
from scratch, it seems that ZODB isn't properly aware of the
_oidsToArticles attribute since it was created in __setstate__ instead
of in __init__. I would really like to know exactly what happens here
as the reason I am utilizing an object database is to be able to
refactor the database with a minimum of fuss, and I would expect
__setstate__ to allow me to make additional attributes in my
instances.
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Help: __getstate__ overriding
I have a Persistent derived class where I want to upgrade from using a
PersistentList to a BTrees.IOBTree.IOBTree.
_articleList is the old Persistent list
_oidsToArticles is the new IOBTree.
def __setstate__(self, state):
articleList = state.get('_articleList')
if articleList:
oidsToArticles = self.makeBTree()
for each in articleList:
oidsToArticles[each.oid()] = each
del state['_articleList']
state['_oidsToArticles'] = oidsToArticles
Persistent.__setstate__(self, state)
This seems to work fine, and I am adding more objects to
_oidsToArticles. The problem is that when I commit the transaction,
nothing is saved. I have tested the same code without __setstate__ and
then it saves just fine. What hoops does one have to jump through to
upgrade the state of a ZODB object?
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] performance tuning of ZODB
Leonardo Rochael Almeida [EMAIL PROTECTED] writes: Hi Syver, Please add this issue to the Collector, including the test (prefereably without the twisted bits) Eh, what is the Collector? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: When should one call Connection.sync?
Dieter Maurer [EMAIL PROTECTED] writes: Syver Enstad wrote at 2004-4-21 18:03 +0200: I have done some experiments with this scheme and I find that everything gets unloaded when do a connection.close() or connection.sync() so that performance takes quite a hit. Maybe, the connection cache is too small. Usually, an incremental cache garbage collection is performed in connection.close() and at transaction boundaries (sync causes an implicit transaction.abort() which means, it marks a transaction boundary). The cache garbage collection tries to flush as many objects from the cache as are necessary to reach the target cache size. Yes that was the problem, I found out about this after some experiments and by upping the cache_size to at least 20 000 I manage to keep the instances in memory even when I do a sync or close. Thank you anyway. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] performance tuning of ZODB
I've used the cache_size paramter to the constructor of the DB to good effect. Are there any further gotchas for ensuring that the ZODB stays in memory as much as possible? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] performance tuning of ZODB
Toby Dickenson [EMAIL PROTECTED] writes: On Thursday 22 April 2004 10:42, Syver Enstad wrote: I've used the cache_size paramter to the constructor of the DB to good effect. Are there any further gotchas for ensuring that the ZODB stays in memory as much as possible? Memory usage scales proportional to the number of threads. Reducing the number of worker threads lets you increase the per-thread cache size. I don't use connection per thread semantics but I think I understand what you mean. I should pass a lower number than the default to the pool_size keyword argument for the DB constructor? What about the other arguments to the DB constructor? cache_deactivate_after sounds interesting. Since I am running ZODB in a web server I don't want the data to timeout and disappear from memory since every object I have should be loaded at all times. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] performance tuning of ZODB
Toby Dickenson [EMAIL PROTECTED] writes:
On Thursday 22 April 2004 11:43, Syver Enstad wrote:
cache_deactivate_after sounds interesting. Since I am running ZODB in
a web server I don't want the data to timeout and disappear from
memory since every object I have should be loaded at all times.
Thats why ZODB ignores that parameter now ;-)
Good :-)
I have a strange case here with ReadConflictErrors. I don't know if
this is covered already but anyway. I am using ZODB 3.2 so this might
be fixed in 3.3 for all I know.
Just replace the twisted stuff with the standard lib unittest module to
have it work on a computer without twisted installed.
import pdb
import sys
import time
import threading
import os
from ZODB import DB
from ZODB.PersistentList import PersistentList
from ZODB.FileStorage import FileStorage
from ZODB.POSException import ReadConflictError
from Persistence import Persistent
from twisted.trial import unittest
class Computer(Persistent):
def __init__(self, oidInteger):
self._oid = oidInteger
self._articleStatus = None
self._endUserPrice = None
self._location = None
def setArticleStatus(self, anObject):
self._articleStatus = anObject
def setEndUserPrice(self, anObject):
self._endUserPrice = anObject
def setLocation(self, anObject):
self._location = anObject
class MockArticleDb(Persistent):
def __init__(self):
self._articles = PersistentList()
def addArticle(self, anArticle):
self._articles.append(anArticle)
def articles(self):
return self._articles
class TestReadConflictError(unittest.TestCase):
def setUp(self):
try:
os.unlink('test.fs')
except OSError, err:
pass
storage = FileStorage(
'test.fs',
create=True)
db = DB(storage, cache_size=10)
connection = db.open()
connection.setLocalTransaction()
connection.root()['articledb'] = MockArticleDb()
computer = Computer(1)
computer.setArticleStatus('For sale')
computer.setEndUserPrice(10050)
computer.setLocation('H0101')
connection.root()['articledb'].addArticle(computer)
connection.getTransaction().commit()
connection.close()
db.close()
storage.close()
storage = FileStorage(
'test.fs',
create=False)
self._db = DB(storage, cache_size=10)
def writeStuff(self, connection):
begin = time.time()
connection.sync()
articleDb = connection.root()['articledb']
eachBegin = time.time()
computer = Computer(1)
computer.setArticleStatus('For sale')
computer.setEndUserPrice(10050)
computer.setLocation('H0101')
articleDb.addArticle(computer)
connection.getTransaction().commit()
print 'write list length', len(articleDb.articles())
def doRead(self, oddRead, connection):
tryCount = 1
articleDb = connection.root()['articledb']
try:
for article in articleDb.articles():
pass
except ReadConflictError, err:
if True:
connection.sync()
else: # this also works
connection.close()
connection = self._db.open()
connection.setLocalTransaction()
for article in articleDb.articles():
pass
tryCount += 1
print 'read list length', len(articleDb.articles())
if oddRead:
# on first, third, fifth read we get a ReadConflictError
self.assertEquals(
2,
tryCount)
else:
# but on second, fourth and so forth we don't
self.assertEquals(
1,
tryCount)
def testProvokeReadConflictError(self):
conn2 = self._db.open()
conn2.setLocalTransaction()
for each in range(1, 11): # 1000 works also but takes a bit of time
begin = time.time()
connection = self._db.open()
connection.setLocalTransaction()
self.writeStuff(conn2)
oddRead = each % 2 != 0
self.doRead(oddRead, connection)
connection.close()
if __name__ == '__main__':
from twisted.scripts.trial import run
import sys
sys.argv.append(sys.argv[0])
run()
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] When should one call Connection.sync?
I am using ZODB 3.2 in a twisted based web application. I have read that I need to call sync to keep the connection up to date. When exactly should I call sync? Are there any drawbacks with calling it immediately after getting a connection, like this: # for each http request. connection = db.open() # (a DB instance) connection.setLocalTransaction() connection.sync() # start using the ZODB here. # if something needs to be committed connection.getTransaction().commit() ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] class methods in Persistent objects
Is it possible to have class methods on ZODB Persistent objects? I am using 3.2 so that means I am using ExtensionClass. I guess that moving to 3.3 would mean being able to use class methods? What is the expected time that the 3.3 release will be ready for production use? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Unique ID generation with ZODB.
Andreas Jung [EMAIL PROTECTED] writes: The simplest solution is to keep the counter as file in the filesystem. Access to the file must be synchronized through a lock object (- Python docs). Just a few lines of Python code. It has to work in windows too, so it's a bit more involved but should be manageable with win32all. I don't particularely like that I have another piece of data to backup though. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Unique ID generation with ZODB.
How does one implement a unique id generator in ZODB. The id's should be strictyl ascending and be a an int. Would opening a temporary connection increment a persistent counter and commit do the trick? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZODB with twisted web.
I am checking out how to use ZODB with twisted web. I thought that I would have the DB instance globally accesible and call the open method to get a connection on each request. I thought that if I use connection.getTransaction().commit() before sending the response I will commit on the connection for that request. Is this necesarily true when using twisted web because it is single threaded and it seems that ZODB finds the correct transaction by checking the thread. I am a bit confused here and would rather like to avoid having one request commit changes for another request. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
