Re: [ZODB-Dev] Weird KeyError with OOBTree
On 16 August 2010 17:29, Pedro Ferreira wrote: > > >> Consider using one >> of these alternatives instead: >> >> * Set the IOTreeSet as an attribute directly on the persistent object. >> > > You mean on the persistent object I am using as key? Yes. >> * Use http://pypi.python.org/pypi/zope.intid and use the intid for the >> key. (This uses http://pypi.python.org/pypi/zope.keyreference which >> uses the referenced object's oid and database name to perform the >> comparison, avoiding the need to load the persistent object.) >> > > This looks really nice. However it seems to depend on a lot of zope > libraries that I'm currently including: location, component, security... > well, I guess they're not that large. I will give it a look, maybe I'll use > it. I guess you could avoid the dependencies by using (obj._p_jar.db().database_name, obj._p_oid) as the key. Laurence ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Weird KeyError with OOBTree
> libraries that I'm currently including: location, component, security... > "libraries that I'm currently *NOT* including" of course :) -- José Pedro Ferreira Indico Team IT-UDS-AVC 513-R-0042 CERN, Geneva, Switzerland ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Weird KeyError with OOBTree
> Consider using one > of these alternatives instead: > > * Set the IOTreeSet as an attribute directly on the persistent object. > You mean on the persistent object I am using as key? > * Use http://pypi.python.org/pypi/zope.intid and use the intid for the > key. (This uses http://pypi.python.org/pypi/zope.keyreference which > uses the referenced object's oid and database name to perform the > comparison, avoiding the need to load the persistent object.) > This looks really nice. However it seems to depend on a lot of zope libraries that I'm currently including: location, component, security... well, I guess they're not that large. I will give it a look, maybe I'll use it. Thanks a lot! Pedro -- José Pedro Ferreira Indico Team IT-UDS-AVC 513-R-0042 CERN, Geneva, Switzerland ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Weird KeyError with OOBTree
On 16 August 2010 13:13, Tres Seaver wrote: > Hanno Schlichting wrote: >> On Mon, Aug 16, 2010 at 12:14 PM, Pedro Ferreira >> wrote: >>> Could this be some problem with using persistent objects as keys in a BTree? >>> Some comparison problem? >> >> I'm not entirely sure about this, but I think using persistent objects >> as keys isn't supported. Looking at the code, I doubt using anything >> expect simple types like unicode strings or tuples of simple types >> will work without further work. >> >>>From what I can see in the code, BTree's use functions like >> PyObject_Compare to compare different keys. Persistent doesn't >> implement any special compare function and falls back to the standard >> hash algorithm for an object. This happens to be its memory address. >> The memory address obviously changes over time and the same address >> gets reused for different objects. >> >> I think implementing a stable hash function for your type could make >> this work though. >> >> The ZODB gods correct me please :) > > Btrees require "comparability", rather than "hashability": your > persistent type needs to define a total ordering[1], which typically > means defining '__cmp__' for your class. You could also define just > '__eq__' and '__lt__', but '__cmp__' is slightly more efficient. > > > [1]http://www.zodb.org/documentation/guide/modules.html#total-ordering-and-persistence While ZODB 3.8 makes it possible to use Persistent objects as keys in a BTree, it's almost certainly a bad idea as a lookup will incur many more object loads while traversing the BTree as the Persistent keys will have to be loaded before they can be compared. Consider using one of these alternatives instead: * Set the IOTreeSet as an attribute directly on the persistent object. * Use http://pypi.python.org/pypi/zope.intid and use the intid for the key. (This uses http://pypi.python.org/pypi/zope.keyreference which uses the referenced object's oid and database name to perform the comparison, avoiding the need to load the persistent object.) Laurence ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Weird KeyError with OOBTree
On Mon, Aug 16, 2010 at 2:04 PM, Pedro Ferreira wrote: > I think implementing a stable hash function for your type could make > this work though. > > From what I read, ZODB doesn't use hash functions, relying on __cmp__ > instead. So, I guess I should make my class non-persistent and implement a > __cmp__ function for it... Right, implementing __cmp__ or all of the rich compare functions would be best. The __hash__ is just used as the default backend for __cmp__ of object. It's probably better to not rely on that indirection and implement compare directly. Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Weird KeyError with OOBTree
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hanno Schlichting wrote: > On Mon, Aug 16, 2010 at 12:14 PM, Pedro Ferreira > wrote: >> Could this be some problem with using persistent objects as keys in a BTree? >> Some comparison problem? > > I'm not entirely sure about this, but I think using persistent objects > as keys isn't supported. Looking at the code, I doubt using anything > expect simple types like unicode strings or tuples of simple types > will work without further work. > >>From what I can see in the code, BTree's use functions like > PyObject_Compare to compare different keys. Persistent doesn't > implement any special compare function and falls back to the standard > hash algorithm for an object. This happens to be its memory address. > The memory address obviously changes over time and the same address > gets reused for different objects. > > I think implementing a stable hash function for your type could make > this work though. > > The ZODB gods correct me please :) Btrees require "comparability", rather than "hashability": your persistent type needs to define a total ordering[1], which typically means defining '__cmp__' for your class. You could also define just '__eq__' and '__lt__', but '__cmp__' is slightly more efficient. [1]http://www.zodb.org/documentation/guide/modules.html#total-ordering-and-persistence Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkxpK18ACgkQ+gerLs4ltQ5gIwCfQDXxLwxpWrghFtRF76MBxesS RtcAnA3+yv+6+XfDRkPBBZcZpoqZ6mO0 =ygVN -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Weird KeyError with OOBTree
PyObject_Compare to compare different keys. Persistent doesn't implement any special compare function and falls back to the standard hash algorithm for an object. This happens to be its memory address. The memory address obviously changes over time and the same address gets reused for different objects. Makes total sense... actually: http://www.zodb.org/documentation/guide/modules.html#total-ordering-and-persistence Seems to indicate that. Actually, i believe that what mislead me was: " Prior to this change, it was not safe to use Persistent objects as keys in a BTree." ( http://docs.zope.org/zope3/Code/ZODB/ConflictResolution.txt/index.html) I think implementing a stable hash function for your type could make this work though. From what I read, ZODB doesn't use hash functions, relying on __cmp__ instead. So, I guess I should make my class non-persistent and implement a __cmp__ function for it... Thanks a lot! Pedro -- José Pedro Ferreira Indico Team IT-UDS-AVC 513-R-0042 CERN, Geneva, Switzerland ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Weird KeyError with OOBTree
On Mon, Aug 16, 2010 at 12:14 PM, Pedro Ferreira wrote: > Could this be some problem with using persistent objects as keys in a BTree? > Some comparison problem? I'm not entirely sure about this, but I think using persistent objects as keys isn't supported. Looking at the code, I doubt using anything expect simple types like unicode strings or tuples of simple types will work without further work. >From what I can see in the code, BTree's use functions like PyObject_Compare to compare different keys. Persistent doesn't implement any special compare function and falls back to the standard hash algorithm for an object. This happens to be its memory address. The memory address obviously changes over time and the same address gets reused for different objects. I think implementing a stable hash function for your type could make this work though. The ZODB gods correct me please :) Hanno ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Weird KeyError with OOBTree
Hello all, I am running out of ideas, so I though maybe someone could help me spot the problem in this one. I have created a minimal example that reproduces the problem - basically, i have an OOBTree where keys are persistent objects and values are IOTreeSets. I am just storing elements, closing the transaction after each operation. Around the 600th element (sometimes 900th, it varies but seems to never reach 1000) a KeyError is thrown, just as if the operation in the previous line hadn't been performed. If i comment out the db instructions, everything runs as expected. This is what I get when I run the code: """ [...] 604 605 606 607 608 609 Traceback (most recent call last): File "test-zodb.py", line 40, in struct.index_obj(obj, i) File "test-zodb.py", line 20, in index_obj print number, self._idx[obj] KeyError: <__main__.DummyObject object at 0x844822c> """ Could this be some problem with using persistent objects as keys in a BTree? Some comparison problem? Any help would be much appreciated :) Thanks, Pedro oobtree.py Description: application/python ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
