Re: [ZODB-Dev] Re: What makes the ZODB slow?
On 26 Jun 2006, at 15:02, Chris Withers wrote: Florent Guillaume wrote: BTrees perform best when keys' prefixes are randomly distributed. So if your application generates keys like 'foo001', 'foo002',... you'll get lots of conflicts. Same for consecutive integers in IOBTree. Tempted to call bullshit on this, since there's code in the catalog to specifically assign series of keys... ...of course, that code may be evil, and people with bigger brains (hi Tim/Jeremy/Jim!) would have to comment.. I can comment, I have a big brain too: the code in the catalog uses per-connection series of keys, so no conflicts arise. Florent -- Florent Guillaume, Nuxeo (Paris, France) Director of RD +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: More exciting ZODB errors ;-)
On 26 Jun 2006, at 15:12, Chris Withers wrote: Florent Guillaume wrote: This patch didn't seem to cure our Zope 2.8 afflicted with the assert end is not None problem. Florent, Have you actually seen any problems from these errors? I still see them occasionally, but they never seem to have any ill effects, other than leaving me feeling uneasy when I get mailed these kind of errors my MailingLogger... The AssertionError makes it back to the user, as it's not a ConflictError. That's a big problem. If anyone has a means to reproduce these errors, I'd gladly instrument the ZODB code to get to the bottom of it and fix it. Florent -- Florent Guillaume, Nuxeo (Paris, France) Director of RD +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: What makes the ZODB slow?
Andreas Jung wrote: BTrees perform best when keys' prefixes are randomly distributed. So if your application generates keys like 'foo001', 'foo002',... you'll get lots of conflicts. Same for consecutive integers in IOBTree. Tempted to call bullshit on this, since there's code in the catalog to specifically assign series of keys... Calm down Oh, sorry, forgot some smilies ;-) Don't worry, perfectly calm here *grinz* Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: getting the object ID (_p_oid attribute)
On Jun 26, 2006, at 3:16 PM, Florent Guillaume wrote: On 26 Jun 2006, at 21:11, Robert Gravina wrote: On 2006/06/27, at 3:49, Benji York wrote: Robert Gravina wrote: I just tried loading a persisted object interactively and noticed that although the _p_oid doesn't print out as anything (and hence I always thought it was empty in my debugging prints), it isn't actually None! Can anyone explain this? (here p is my persisted object) p._p_oid '\x00\x00\x00\x00\x00\x00\x00\x08' print p._p_oid p._p_oid is None False What would you expect to see if you printed out seven null characters and a backspace? Hahaha - that's a good point! I was expecting IDs to look, well, something like asdf23asdf. Well, anyway thankyou! I seemed to have solved this problem. I was able to write a __eq__ function like this: def __eq__(self, other): if isinstance(other, name of my class): if hasattr(other,_p_oid) and other._p_oid != None and (other._p_oid == self._p_oid): return True else: return False and now can compare objects for equality after the (Twisted) client edits them and sends them back. You can shorten that to def __eq__(self, other): return aq_base(self) is aq_base(other) Heh, good point. And you can ditch the aq_base if you don't use acquisition-based classes. (or don't have proxies generally--Zope 3 has a number of them, for instance.) (this wouldn't work if you had the same persistent object from different connections...but don't go there!) Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: getting the object ID (_p_oid attribute)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Robert Gravina wrote: On 2006/06/27, at 3:49, Benji York wrote: Robert Gravina wrote: I just tried loading a persisted object interactively and noticed that although the _p_oid doesn't print out as anything (and hence I always thought it was empty in my debugging prints), it isn't actually None! Can anyone explain this? (here p is my persisted object) p._p_oid '\x00\x00\x00\x00\x00\x00\x00\x08' print p._p_oid p._p_oid is None False What would you expect to see if you printed out seven null characters and a backspace? Hahaha - that's a good point! I was expecting IDs to look, well, something like asdf23asdf. Well, anyway thankyou! I seemed to have solved this problem. I was able to write a __eq__ function like this: def __eq__(self, other): if isinstance(other, name of my class): if hasattr(other,_p_oid) and other._p_oid != None and (other._p_oid == self._p_oid): return True else: return False and now can compare objects for equality after the (Twisted) client edits them and sends them back. Anyway, thanks again. Note that 'hasattr' swallows *all* exceptions, including some which might not be appropriate. I think I would recommend: if isinstance(other, my class): o_jar = getattr(other, '_p_jar', None) o_oid = getattr(other, '_p_oid', None) return ( o_jar and o_oid and o_jar == self._p_jar and o_oid == self._p_oid) since OIDs are guaranteed to be unique only within the scope of a given database (the '_p_jar' is a connection to the database). Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEoDQU+gerLs4ltQ4RAnrPAKCuR/ZL+btXAzz6tQ/TPhycLbGM5gCglPXB f/O+EhSS2QgfEyikJQi7wqA= =meKl -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: More exciting ZODB errors ;-)
Hallo Florent, Florent Guillaume wrote at 2006-6-26 00:53 +0200: Dieter Maurer wrote: Chris Withers wrote at 2005-12-14 16:23 +: ... File lib/python/ZODB/Connection.py, line 788, in _setstate_noncurrent assert end is not None AssertionError This means that the latest modification for this object lies before the respective transaction. In this case, we should not have an invalidation for the object, such that we would not call _setstate_current. I expect a missing flush_invalidations during Connection._setDB. I had to add such a call in our ZODB version: ... This patch didn't seem to cure our Zope 2.8 afflicted with the assert end is not None problem. Could you expand a bit on why you think it would matter? From what I understand of the code, it would only come into play during a refresh() operation. You are right. My modification only makes refresh safer. Could you expand on the it does more than cache handling? Invalidations being synchronous, It also resets connection._invalidated and connection._txn_time. BTW do you have other patches in your local patched ZODB? I'm willing to look at all of them and see if they can be integrated. Lot's of them (but not related to this problem which I haven't seen so far) -- and not all of them working already. I could make a tarball from the ZODB parts but I doubt that you would be very happy with it... -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: getting the object ID (_p_oid attribute)
def __eq__(self, other): if isinstance(other, name of my class): if hasattr(other,_p_oid) and other._p_oid != None and (other._p_oid == self._p_oid): return True else: return False and now can compare objects for equality after the (Twisted) client edits them and sends them back. You can shorten that to def __eq__(self, other): return aq_base(self) is aq_base(other) Heh, good point. I assume you're referring to repacing just isinstance(other, name of my class)? Actually, to be honest I don't even know what Acquisition based classes are, so I'm probably not using them. I'm just subclassing Persistent And you can ditch the aq_base if you don't use acquisition-based classes. (or don't have proxies generally--Zope 3 has a number of them, for instance.) (this wouldn't work if you had the same persistent object from different connections...but don't go there!) Don't worry, I'm happy enough just going this far! Thanks all for your great and fast help! I've struggled with this on and off for a couple of days (and it's 4am).. what a relief! Robert ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] What makes the ZODB slow?
[Dieter Maurer] The newest pickle formats can also handle the class references is bit more efficiently -- at least when a single transaction modifies many objects of the same class. [Chris Withers] I know ZC was involved in the work to introduce these new pickle formats, but are they actually used in ZODB yet? [Dieter] I think the optimization I refered to (if you have several instances of the same class in a pickle, then they all share a single instance of the class name and reference this one) is already used. I'm not sure what you have in mind there. It's always been true that pickle was _able_ to reuse common bits, but this is effectively disabled in ZODB on a cross-persistent-object basis by (from a recent serialize.py): def _dump(self, classmeta, state): # To reuse the existing cStringIO object, we must reset # the file position to 0 and truncate the file after the # new pickle is written. self._file.seek(0) self._p.clear_memo() self._p.dump(classmeta) self._p.dump(state) self._file.truncate() return self._file.getvalue() The self._p.clear_memo() there makes the pickler forget everything it's done, so that the pickle for a persistent object is self-contained. For example, if you store an OOBTree whose internal state contans 100 OOBuckets, the string BTrees._OOBucket appears 100 times in the data record, and string OOBTree even more. Jeremy once analyzed a customer Data.fs and incidentally discovered that about half the space was consumed by repetitions of such BTree-related strings; no idea whether that's typical, although I wouldn't be surprised if it were. An entirely new gimmick was introduced in pickle protocol 2, the extension registry described in PEP 307: http://www.python.org/dev/peps/pep-0307/ That _allows_ an application to register popular module and class string names that pickles can reference later via teensy 2- or 3-byte (independent of string length) opcodes. In effect, such strings are stored in the _implementation_ of pickle instead of inside pickles. AFAIK, nobody anywhere has used this yet, outside of Python's test suite. It was intended to be a simple, cheap approach to cutting pickle bloat for apps motivated enough to set up the registry. You'll note that half the one-byte codes are reserved for Zope :-) You mean an optimization to make the pickle size for some new style classes smaller. That's not yet used because it could make the storage exchange between different Python versions impossible (the older Python versions would not understand the new pickle protocol). The need for protocol 2 is also why the extension registry (above) can't be used so long as older Pythons are in the mix. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] What makes the ZODB slow?
On Mon, Jun 26, 2006 at 05:28:51PM -0400, Tim Peters wrote: | AFAIK, nobody anywhere has used this yet, outside of Python's test | suite. It was intended to be a simple, cheap approach to cutting | pickle bloat for apps motivated enough to set up the registry. You'll | note that half the one-byte codes are reserved for Zope :-) | | You mean an optimization to make the pickle size for some | new style classes smaller. That's not yet used because it could | make the storage exchange between different Python versions impossible | (the older Python versions would not understand the new pickle protocol). | | The need for protocol 2 is also why the extension registry (above) | can't be used so long as older Pythons are in the mix. Does that mean that if someone didn't care about older python's in the mix and were willing to register those shorter byte code extensions for it's own Zope that person would likely see great improvements in pickle size reduction, and that it would even improve ZEO transport by reducing the size of the data that is transferred? Maybe we should propose such changes to Zope 2.10/2.11 since it already requires Python 2.4 (since Zope 2.9). -- Sidnei da Silva Enfold Systemshttp://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
