Re: [ZODB-Dev] What's best to do when there is a failure in the second phase of 2-phase commit on a storage server
Jim Fulton wrote at 2008-10-1 13:40 -0400: ... It may well be that a restart *may* not lead into a fully functional state (though this would indicate a storage bug) A failure in tpc_finish already indicates a storage bug. Maybe -- although file system is full might not be so easy to avoid in all cases -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] What's best to do when there is a failure in the second phase of 2-phase commit on a storage server
Christian Theune wrote at 2008-10-3 10:32 +0200: On Fri, 2008-10-03 at 09:55 +0200, Dieter Maurer wrote: Jim Fulton wrote at 2008-10-1 13:40 -0400: ... It may well be that a restart *may* not lead into a fully functional state (though this would indicate a storage bug) A failure in tpc_finish already indicates a storage bug. Maybe -- although file system is full might not be so easy to avoid in all cases That should be easy to avoid by allocating the space you need in the first phase and either release it on an abort or write your 'committed' marker into it in the second phase. That's true for a FileStorage -- but it may not be that easy for other storages (e.g. BSDDB storage). -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] What's best to do when there is a failure in the second phase of 2-phase commit on a storage server
On Fri, 2008-10-03 at 10:51 +0200, Dieter Maurer wrote: That's true for a FileStorage -- but it may not be that easy for other storages (e.g. BSDDB storage). Those storages using another system in the backend have to rely on them providing a two-phase commit API which needs to implement the second phase guarantee, or not? -- Christian Theune · [EMAIL PROTECTED] gocept gmbh co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Object uniqueness
Hi! In which scope does zodb preserve object uniqueness? The problem I'm having is I have Person-objects in two places, in a set of persons (root[Person], which is an OOSet) and in name-index (root[(Name, Person)], which is a OOBTree). Now if I find people by name from the name index, they seem not to be the same in the set of all Person-objects... All the Persons get into the OSet and OOBTree like this: firstly the created person is just added to the OOSet; then its index method is called which returns a list of tuples -- in my case it's just (from the objects' point of view) [(self.name, self)]. USing that my db gets the types of both of them-- Name and Person --and then puts them in the appropriate index by doing root[(Name, Person)][ name ].update([ prerson ]). And now if I retrieve some person from the name-index, I cannot remove (by method remove) them from the OOSet of all persons, because it raises a KeyError. Yet if I iter through the set of persons, one of them *is* the one I got from the name-index... What am I thinking wrongly about? Markus ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Broken instances after refactoring in ZODB
Hi, I wonder if anyone can help me. I've got a custom Python cgi which runs in one directory using ZODB for the database. It runs in a directory off the site root ie. /System It's got a bit unwieldy in there so I wanted to separate the core modules out as I don't really touch them any more, so adding subdirectories: /System/Core /System/Application In System, I created a dummy frontend CGI like the following: # /System/frontend.py import ZODB from Core import * from Application import * site = Start.ProcessRequest() # Start is a module in Core site.Main() For some reason, all the Persistent derived objects in the database are returning: broken Module.Class instance Does ZODB do something with __import__ modules which may cause it to lose the references to the classes/modules? Is there any way to fix the database or do I have to return to putting all my .py files in one directory. Many thanks, Adam ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] What's best to do when there is a failure in the second phase of 2-phase commit on a storage server
On Oct 3, 2008, at 3:55 AM, Dieter Maurer wrote: Jim Fulton wrote at 2008-10-1 13:40 -0400: ... It may well be that a restart *may* not lead into a fully functional state (though this would indicate a storage bug) A failure in tpc_finish already indicates a storage bug. Maybe -- although file system is full might not be so easy to avoid in all cases This is why FileStorage writes data to the end of the file before returning from tpc_vote. tpc_finish only updates a single byte of the file. This is why FileStorage no-longer tries to write it's index during tpc_finish. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Object uniqueness
I think I know now -- objects compare identical and equal only if they are
in the same container object. (Check the code below)
I don't mean to be negative, but why can't ZODB be like a big RAM? Like
when i store an object to multiple places on the same DB, then it would
actually be the same in each place. (Like if you comment out the Commit,
close and reload part of the below code, all the comparisons evaluate to
True)
Markus
# -*- coding: cp1252 -*-
# ZODB
from ZODB import FileStorage, DB
import transaction
# Trees
from BTrees.IOBTree import IOBTree as IO
from BTrees.OOBTree import OOBTree as OO, OOSet as OSet
# Open db
storage = FileStorage.FileStorage('test.fs')
db = DB(storage)
conn = db.open()
root = conn.root()
# The one and only class
class X(object):
attr1 = 1
def __init__(self, attr2):
self.attr2 = attr2
# Initiate three containers, OOBTree and OOSet
root[0] = OO()
root[1] = OSet()
root[0]['subBTree'] = OO()
# The only object to be stored...
x = X('abc')
# ...and storing it
root[0]['a'] = x
root[0]['b'] = x
root[0]['subBTree']['a'] = x
root[1].update([ x ])
# Commit, close and reload
transaction.commit()
db.close()
storage = FileStorage.FileStorage('test.fs')
db = DB(storage)
conn = db.open()
root = conn.root()
# Shorthands...
a = root[0]['a']
asub = root[0]['subBTree']['a']
b = root[0]['b']
c = iter(root[1]).next()
# ...and the comparison
print 'a is asub:', a is asub
print 'a == asub:', a == asub
print 'a is b:', a is b
print 'a == b:', a == b
print 'a is c:', a is c
print 'a == c:', a == c
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Object uniqueness
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Markus Läll wrote: I think I know now -- objects compare identical and equal only if they are in the same container object. (Check the code below) I don't mean to be negative, but why can't ZODB be like a big RAM? Like when i store an object to multiple places on the same DB, then it would actually be the same in each place. (Like if you comment out the Commit, close and reload part of the below code, all the comparisons evaluate to True) If your classes derive from persistence.Persistent, then their instances' identities will be preserved across serializations. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFI5kb7+gerLs4ltQ4RAq3yAJ9USY6EX+cssrxT0ZOOlS6EG+OREwCfYn5g MqQBgyaT2JnWYe18jJSI/pE= =Z0o3 -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Broken instances after refactoring in ZODB
On Fri, Oct 03, 2008 at 11:37:35AM +0100, Adam wrote: Hi, I wonder if anyone can help me. I've got a custom Python cgi which runs in one directory using ZODB for the database. It runs in a directory off the site root ie. /System It's got a bit unwieldy in there so I wanted to separate the core modules out as I don't really touch them any more, so adding subdirectories: /System/Core /System/Application In System, I created a dummy frontend CGI like the following: # /System/frontend.py import ZODB from Core import * from Application import * site = Start.ProcessRequest() # Start is a module in Core site.Main() For some reason, all the Persistent derived objects in the database are returning: broken Module.Class instance Does ZODB do something with __import__ modules which may cause it to lose the references to the classes/modules? The ZODB stores fully qualified class names (package.module.class). If you move the modules around, ZODB can no longer find the classes in their old locations. Is there any way to fix the database Yes, but it's not pretty: 1. create modules in their old locations that import all the classes from the new locations (and do nothing else). 2. write a script that opens the DB, iterates over all persistent objects in it, touches every single one of them (obj._p_activate() followed by obj._p_changed = True) and commits the transaction. If you've done everything right, now the database should refer to all the classes using their new locations. or do I have to return to putting all my .py files in one directory. Third alternative: create modules for backwards compatibility (just the step 1 described above). Marius Gedminas -- Those who can't write, write manuals. signature.asc Description: Digital signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] conflict errors in persistent ZEO client cache
On Sep 25, 2008, at 6:31 AM, Jim Fulton wrote: On Sep 25, 2008, at 1:12 AM, Jeff Shell wrote: In ZODB 3.8.0, while prepping a customer's site to go from staging to production and doing a fair amount of big operations, we started seeing conflict errors. It was a 200MB cache size, with a persistent client cache. Keeping the same cache size (against a base storage of about 700MB) but turning off the persistent client cache seemed to make the problem go away. The persistent cache file at the time of the conflict error was close to 200MB in size, but doesn't it always go to the size configured, regardless of how much data is inside? Since ZODB 3.8.0, I have fixed *many* ZEO cache bugs. What you describe is one symptom. I suggest upgrading to 3.8.1b8. Just to follow up and hopefully help get 3.8.1 out the door - trying these sites and configurations on 3.8.1b8 worked much better than on 3.8.0. Thanks, Jeff Shell [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] conflict errors in persistent ZEO client cache
On Oct 3, 2008, at 2:34 PM, Jeff Shell wrote: On Sep 25, 2008, at 6:31 AM, Jim Fulton wrote: On Sep 25, 2008, at 1:12 AM, Jeff Shell wrote: In ZODB 3.8.0, while prepping a customer's site to go from staging to production and doing a fair amount of big operations, we started seeing conflict errors. It was a 200MB cache size, with a persistent client cache. Keeping the same cache size (against a base storage of about 700MB) but turning off the persistent client cache seemed to make the problem go away. The persistent cache file at the time of the conflict error was close to 200MB in size, but doesn't it always go to the size configured, regardless of how much data is inside? Since ZODB 3.8.0, I have fixed *many* ZEO cache bugs. What you describe is one symptom. I suggest upgrading to 3.8.1b8. Just to follow up and hopefully help get 3.8.1 out the door - trying these sites and configurations on 3.8.1b8 worked much better than on 3.8.0. Great. Thanks. Hopefully, after fixing a couple of spurious windows test failures, I'll get 3.8.1 out this weekend. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Broken instances after refactoring in ZODB
Marius Gedminas wrote: 1. create modules in their old locations that import all the classes from the new locations (and do nothing else). 2. write a script that opens the DB, iterates over all persistent objects in it, touches every single one of them (obj._p_activate() followed by obj._p_changed = True) and commits the transaction. Marius said it correctly, but I'd like to emphasize that it's not enough to touch just the objects whose class changed, since persistent references encode the module and class name. It's best to just touch everything. You will have to use multiple transactions to touch everything in a large database. Shane ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
