Re: [ZODB-Dev] Different weird errors with Zope 2.8.4
Yay! Where's the correct place to report these nowadays? cheers, Chris Dieter Maurer wrote: Chris Withers wrote at 2005-11-29 14:33 +: Hot on the heals of my posts about Shouldn't load state for comes the eagerly awaited sequel Couldn't load state for ;-) ... File lib/python/ZEO/zrpc/connection.py, line 531, in call r_flags, r_args = self.wait(msgid) File lib/python/ZEO/zrpc/connection.py, line 648, in wait self.replies_cond.release() File lib/python2.3/threading.py, line 113, in release assert self.__owner is me, release() of un-acquire()d lock AssertionError: release() of un-acquire()d lock Looks like a ZEO bug (more precisely a ZEO.zrpc.conntection bug). Probably a race condition. -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Re: [Zope] DateTime mess
Sorry, I couldn't find a comprehensible question here after reasonable effort to extract one. Clearly, Zope2's DateTime.DateTime.DateTime objects are neither persistent nor do they define any mutating methods. Are those relevant? If not, try to ask a question directly, without assuming everyone has read the other 55 messages in the zope-dev thread ;-) -Original Message- From: Chris Withers [EMAIL PROTECTED] Sent: Wednesday, November 30, 2005 2:03 PM To: Lennart Regebro Cc: zodb-dev@zope.org Subject: [ZODB-Dev] Re: [Zope] DateTime mess Lennart Regebro wrote: On 11/29/05, Chris Withers [EMAIL PROTECTED] wrote: Hmm... how so? I've always thought it quite nice that when, for example, you store the modification time of an object in a DateTime, you can safely update it without worrying about the whole object having to be repickled when you change it... Oh, so that make a difference? datetime objects are unchangeable, Really? didn't know that... so everytime you change it you will have to set the attribute, and I thought the whole object got pickled then, but I guess that it doesn't, then. Honestly, I don't know. Changing to the zodb-dev list so someone who does might comment... cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Different weird errors with Zope 2.8.4
[Chris Withers] Yay! Where's the correct place to report these nowadays? Same as always: ZODB bugs should be reported on a Zope Collector, with topic Database. It's fine to discuss them on zodb-dev, but when (as appears to be the case with this one) nobody has an immediate answer or time to dig into it, opening a collector issue is the way to proceed. If it's only in the mailing list archives, it will be forgotten by all within a week (although it may cheer someone with the same problem 3 years from now when they happen to stumble into the old message while doing a web search ;-)). ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Re: [Zope] DateTime mess
[Jürgen Herrmann] the question was wether DateTime instances (of the new implementation, which is yet to be coded) should mixin Persistent. OK. Since ZODB doesn't care whether you do, is there confusion about what ZODB may or may not do in either case? That is, what's the ZODB issue here? Have you looked at what Zope3 does? I believe it uses Python's new(ish) datetime.datetime objects as a base, and those aren't persistent either. In Zope2-land, a DateTime.DateTime is a large object, and you may have _wanted_ to make them persistent just so they could lose most of their memory-hogging state when ghostified. The state for a datetime.datetime object is much smaller, and I know Gary Poster gave a lot of thought to making pickles for the timezone info in Zope3 efficient too. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope] DateTime mess
Tim Peters wrote: Sorry, I couldn't find a comprehensible question here after reasonable effort to extract one. Clearly, Zope2's DateTime.DateTime.DateTime objects are neither persistent nor do they define any mutating methods. Are those relevant? If not, try to ask a question directly, without assuming everyone has read the other 55 messages in the zope-dev thread ;-) Sorry, my question was that if DateTime's were persistent, would the following code result in a complete pickle for 'a' being written on the second transaction commit? a.someTime = DateTime() get_transaction().commit() ...wait/do stuff... a.someTime = DateTime() ...have we just committed a pickle containing all of 'a'?... Does mixing persistence into DateTime make a difference here? cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope] DateTime mess
On Dec 1, 2005, at 11:38 AM, Tim Peters wrote: ... I know Gary Poster gave a lot of thought to making pickles for the timezone info in Zope3 efficient too. For some definition of a lot of thought. :-) The pickle for pytz.utc is now relatively small (though still adds a non-trivial percentage addition--30%ish?--to a naive datetime IIRC). That's as far as that bit goes. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope] DateTime mess
On Dec 1, 2005, at 11:39 AM, Chris Withers wrote: Sorry, my question was that if DateTime's were persistent, would the following code result in a complete pickle for 'a' being written on the second transaction commit? a.someTime = DateTime() get_transaction().commit() wait/do stuff... a.someTime = DateTime() have we just committed a pickle containing all of 'a'?... Not quite. But if you call commit() again, you do indeed save a new pickle of a. Does mixing persistence into DateTime make a difference here? Not with respect to whether or not a second commit contains a pickle of a. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Re: [Zope] DateTime mess
[Chris Withers] Sorry, my question was that if DateTime's were persistent, would the following code result in a complete pickle for 'a' being written on the second transaction commit? There is only one commit in the following, so I'll assume you intended a second commit at the end: a.someTime = DateTime() get_transaction().commit() Is `a` persistent? I'm assuming that it is. ...wait/do stuff... a.someTime = DateTime() I'm assuming another get_transaction().commit() was intended here. ...have we just committed a pickle containing all of 'a'?... If `a` is persistent, yes. Does mixing persistence into DateTime make a difference here? No. Assuming `a` is persistent, you changed `a`, so `a`'s state gets written out. This really has nothing to do with DateTime. It would be the same answer had you done, e.g., a.someTime = None or a.someTime = hi! or a.someTime = OOBTree() instead. One part is different: if DateTime is not persistent, then the pickle for a's state includes the full state of a.someTime's DateTime attribute (and this is true for any attribute of any non-persistent type). But if DateTime is persistent, then a.someTime is pickled as a distinct object, and the state for `a` contains only a reference to that distinct pickle (and again this is true for any attribute of any persistent type). a's full state is pickled regardless. The difference is in whether a's full state embeds a.someTime's state directly, or points to a distinct pickle for a.someTime's state. If the same DateTime object is referenced by many persistent objects, there can be major storage efficiencies is making DateTime persistent, because the many other objects can all point to the same pickle. I don't know, but I doubt that's a likely scenario for DateTime objects. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Re: [Zope] DateTime mess
[Gary Poster] For some definition of a lot of thought. :-) The pickle for pytz.utc is now relatively small (though still adds a non-trivial percentage addition--30%ish?--to a naive datetime IIRC). That's as far as that bit goes. A naïve datetime has an extraordinarily small state, though, so 30% isn't much in absolute terms. IIRC, we're talking dozen of bytes versus hundreds of bytes for a Zope2 DateTime.DateTime. Note that we have yet to use a new strategy for shrinking pickle sizes: a few years ago Python's pickle code grew support for extension codes, a registry of class/type names that _can_ be referenced by short (as short as 2 bytes) new pickle codes, instead of embedding the module and class name into every pickle, over and over again. I don't recall the exact numbers numbers, but some years ago Jeremy analyzed a customer Data.fs, and discovered that at least half of it consisted of repetitions of the string BTrees.OOBTree.OOBTree ;-) That's the kind of thing the extension code pickle mechanism was intended to address; it's a simple and cheap compression gimmick, but so far unused. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope] DateTime mess
On Dec 1, 2005, at 12:04 PM, Tim Peters wrote: Note that we have yet to use a new strategy for shrinking pickle sizes: a few years ago Python's pickle code grew support for extension codes, a registry of class/type names that _can_ be referenced by short (as short as 2 bytes) new pickle codes, instead of embedding the module and class name into every pickle, over and over again. I don't recall the exact numbers numbers, but some years ago Jeremy analyzed a customer Data.fs, and discovered that at least half of it consisted of repetitions of the string BTrees.OOBTree.OOBTree ;-) That's the kind of thing the extension code pickle mechanism was intended to address; it's a simple and cheap compression gimmick, but so far unused. Yes, I remembered this, and just refreshed my memory. This is the last mention I see in the archives as to ZODB use of protocol 2 (i.e., it doesn't, and prior to Py 2.3.4 it couldn't). http://mail.zope.org/pipermail/zodb-dev/2004-December/008259.html Is that still accurate--that is, does ZODB still not use protocol 2? Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Re: [Zope] DateTime mess
... ...have we just committed a pickle containing all of 'a'?... If `a` is persistent, yes. If not? Then get_transaction().commit() presumably doesn't do much of anything, since no persistent object was changed. It's like asking what this does: i = 2+3 get_transaction().commit() Well, probably nothing (of any interest). ... One part is different: if DateTime is not persistent, then the pickle for a's state includes the full state of a.someTime's DateTime attribute (and this is true for any attribute of any non-persistent type). But if DateTime is persistent, then a.someTime is pickled as a distinct object, and the state for `a` contains only a reference to that distinct pickle (and again this is true for any attribute of any persistent type). a's full state is pickled regardless. The difference is in whether a's full state embeds a.someTime's state directly, or points to a distinct pickle for a.someTime's state. Ah, okay, so having DateTime sublcass Persistent would only really matter if a had _lots_ of DateTime attributes. Matter to what, specifically? I don't see the connection. If many persistent objects reference the _same_ DateTime object (identically the same: d1 is d2 in Python terms, not merely d1 == d2), then making DateTime persistent would allow those objects' states all to refer to a single physical pickle for that DateTime object. That has nothing to do with the sheer number of DateTime attributes, though, it has entirely to do with how often the app can and does force DateTime attributes to reference identically-same DateTime objects. If there are a large number of DateTime attributes that are rarely referenced, then there can be a different reason to want DateTime to be persistent: ZODB creates ghosts for attributes of persistent types when an object is loaded, and if a persistent attribute is never referenced then ZODB never needs to fetch its state or consume RAM to materialize its state. In contrast, attributes of non-persistent types are fully materialized when the containing object is loaded. Does this ever happen? Don't know but I doubt it's common. It's plausible, e.g., that you've got an app managing, say, the open/closed schedules for buildings on a campus, and want to say that each building closes at 8pm 31 Dec and doesn't reopen until 8am 7 Jan. Then the buildings _could_ share the same DateTime objects representing the endpoints of their common schedule that week. Anyone contorting their code to force that to happen is probably insane, though ;-) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Re: [Zope] DateTime mess
[Gary Poster, on pickle extension codes] Yes, I remembered this, and just refreshed my memory. This is the last mention I see in the archives as to ZODB use of protocol 2 (i.e., it doesn't, and prior to Py 2.3.4 it couldn't). http://mail.zope.org/pipermail/zodb-dev/2004-December/008259.html Is that still accurate--that is, does ZODB still not use protocol 2? Afraid so, yes. All current ZODBs still use proto 1. Given that Python 2.4 is about to be required for the next batch of to-be-current Zopes, and there's little ZEO compatibility between pre-3.3 ZODB and post-3.3 ZODB anyway, someone in the community may well want to try s/1/2/g on the ZODB code base and see what happens 0.7 wink. A lot of hard work went into proto 2: http://www.python.org/peps/pep-0307.html and it's really a shame it isn't being used (especially since Zope Corp paid for this development). I expect Zope3 uses new style classes more heavily than Zope2, and proto 2 has major improvements for new-style class instance pickling independent of the extension-code gimmick. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
