Re: [ZODB-Dev] Re: [Zope] DateTime mess
On 12/2/05, Lennart Regebro <[EMAIL PROTECTED]> wrote: > On 12/1/05, Chris Withers <[EMAIL PROTECTED]> wrote: > > Ah, okay, so having DateTime sublcass Persistent would only really > > matter if a had _lots_ of DateTime attributes. Does this ever happen? > > Well, in a calendar each event has at least one. And you can have > quite many events. But on the other hand, if this ever became an > issue, it would (in any good architecture) be easy to substitute the > standard datetime for a *cough* *Ahem* If this ever became an issue it would (in any good architecture) be easy to substitute the standard datetime for a persistent one in these applications. Or, more to the point: Ya Aint Gonna Need It. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope] DateTime mess
On 12/1/05, Chris Withers <[EMAIL PROTECTED]> wrote: > Ah, okay, so having DateTime sublcass Persistent would only really > matter if a had _lots_ of DateTime attributes. Does this ever happen? Well, in a calendar each event has at least one. And you can have quite many events. But on the other hand, if this ever became an issue, it would (in any good architecture) be easy to substitute the standard datetime for a > > > If the same DateTime object is referenced by many persistent objects, there > > can be major storage efficiencies is making DateTime persistent, because the > > "many other objects" can all point to the same pickle. I don't know, but I > > doubt that's a likely scenario for DateTime objects. > > Indeed :-/ Oh well... > > Chris > > -- > Simplistix - Content Management, Zope & Python Consulting > - http://www.simplistix.co.uk > ___ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zodb-dev > -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Re: [Zope] DateTime mess
[Gary Poster, on pickle "extension codes"] > Yes, I remembered this, and just refreshed my memory. This is the last > mention I see in the archives as to ZODB use of protocol 2 (i.e., it > doesn't, and prior to Py 2.3.4 it couldn't). > > http://mail.zope.org/pipermail/zodb-dev/2004-December/008259.html > > Is that still accurate--that is, does ZODB still not use protocol 2? Afraid so, yes. All current ZODBs still use proto 1. Given that Python 2.4 is about to be required for the next batch of to-be-current Zopes, and there's little ZEO compatibility between pre-3.3 ZODB and post-3.3 ZODB anyway, someone in the community may well want to try s/1/2/g on the ZODB code base and see what happens <0.7 wink>. A lot of hard work went into proto 2: http://www.python.org/peps/pep-0307.html and it's really a shame it isn't being used (especially since Zope Corp paid for this development). I expect Zope3 uses "new style" classes more heavily than Zope2, and proto 2 has major improvements for new-style class instance pickling independent of the extension-code gimmick. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Re: [Zope] DateTime mess
... >>> ...have we just committed a pickle containing all of 'a'?... >> If `a` is persistent, yes. > If not? Then get_transaction().commit() presumably doesn't do much of anything, since no persistent object was changed. It's like asking what this does: i = 2+3 get_transaction().commit() Well, "probably nothing (of any interest)". ... >> One part is different: if DateTime is not persistent, then the pickle >> for a's state includes the full state of a.someTime's DateTime attribute >> (and this is true for any attribute of any non-persistent type). But if >> DateTime is persistent, then a.someTime is pickled as a distinct object, >> and the state for `a` contains only a reference to that distinct pickle >> (and again this is true for any attribute of any persistent type). a's >> full state is pickled regardless. The difference is in whether a's full >> state embeds a.someTime's state directly, or "points to" a distinct >> pickle for a.someTime's state. > Ah, okay, so having DateTime sublcass Persistent would only really matter > if a had _lots_ of DateTime attributes. Matter to what, specifically? I don't see the connection. If many persistent objects reference the _same_ DateTime object (identically the same: "d1 is d2" in Python terms, not merely "d1 == d2"), then making DateTime persistent would allow those objects' states all to refer to a single physical pickle for that DateTime object. That has nothing to do with the sheer number of DateTime attributes, though, it has entirely to do with how often the app can and does force DateTime attributes to reference identically-same DateTime objects. If there are a large number of DateTime attributes that are rarely referenced, then there can be a different reason to want DateTime to be persistent: ZODB creates "ghosts" for attributes of persistent types when an object is loaded, and if a persistent attribute is never referenced then ZODB never needs to fetch its state or consume RAM to materialize its state. In contrast, attributes of non-persistent types are fully materialized when the containing object is loaded. > Does this ever happen? Don't know but I doubt it's common. It's plausible, e.g., that you've got an app managing, say, the open/closed schedules for buildings on a campus, and want to say that each building closes at 8pm 31 Dec and doesn't reopen until 8am 7 Jan. Then the buildings _could_ share the same DateTime objects representing the endpoints of their common schedule that week. Anyone contorting their code to force that to happen is probably insane, though ;-) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope] DateTime mess
On Dec 1, 2005, at 12:04 PM, Tim Peters wrote: Note that we have yet to use a new strategy for shrinking pickle sizes: a few years ago Python's pickle code grew support for "extension codes", a registry of class/type names that _can_ be referenced by short (as short as 2 bytes) new pickle codes, instead of embedding the module and class name into every pickle, over and over again. I don't recall the exact numbers numbers, but some years ago Jeremy analyzed a customer Data.fs, and discovered that at least half of it consisted of repetitions of the string "BTrees.OOBTree.OOBTree" ;-) That's the kind of thing the "extension code" pickle mechanism was intended to address; it's a simple and cheap compression gimmick, but so far unused. Yes, I remembered this, and just refreshed my memory. This is the last mention I see in the archives as to ZODB use of protocol 2 (i.e., it doesn't, and prior to Py 2.3.4 it couldn't). http://mail.zope.org/pipermail/zodb-dev/2004-December/008259.html Is that still accurate--that is, does ZODB still not use protocol 2? Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope] DateTime mess
On 12/1/05, Tim Peters <[EMAIL PROTECTED]> wrote: > [Jürgen Herrmann] > > the question was wether DateTime instances (of the new implementation, > > which is yet to be coded) should mixin Persistent. > > OK. Since ZODB doesn't care whether you do Well, I think that was the question. Does ZODB care? :) -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/ ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope] DateTime mess
Tim Peters wrote: There is only one commit in the following, so I'll assume you intended a second commit at the end: yes, *sigh*, must engage brain... a.someTime = DateTime() get_transaction().commit() Is `a` persistent? I'm assuming that it is. Yes. ...wait/do stuff... a.someTime = DateTime() I'm assuming another get_transaction().commit() was intended here. Yes. ...have we just committed a pickle containing all of 'a'?... If `a` is persistent, yes. If not? Does mixing persistence into DateTime make a difference here? No. Assuming `a` is persistent, you changed `a`, so `a`'s state gets written out. This really has nothing to do with DateTime. It would be the same answer had you done, e.g., a.someTime = None or a.someTime = "hi!" or a.someTime = OOBTree() instead. *sigh* thought as much :-S One part is different: if DateTime is not persistent, then the pickle for a's state includes the full state of a.someTime's DateTime attribute (and this is true for any attribute of any non-persistent type). But if DateTime is persistent, then a.someTime is pickled as a distinct object, and the state for `a` contains only a reference to that distinct pickle (and again this is true for any attribute of any persistent type). a's full state is pickled regardless. The difference is in whether a's full state embeds a.someTime's state directly, or "points to" a distinct pickle for a.someTime's state. Ah, okay, so having DateTime sublcass Persistent would only really matter if a had _lots_ of DateTime attributes. Does this ever happen? If the same DateTime object is referenced by many persistent objects, there can be major storage efficiencies is making DateTime persistent, because the "many other objects" can all point to the same pickle. I don't know, but I doubt that's a likely scenario for DateTime objects. Indeed :-/ Oh well... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Re: [Zope] DateTime mess
[Gary Poster] > For some definition of "a lot of thought". :-) The pickle for pytz.utc > is now relatively small (though still adds a non-trivial percentage > addition--30%ish?--to a naive datetime IIRC). That's as far as that bit > goes. A naïve datetime has an extraordinarily small state, though, so 30% isn't much in absolute terms. IIRC, we're talking dozen of bytes versus hundreds of bytes for a Zope2 DateTime.DateTime. Note that we have yet to use a new strategy for shrinking pickle sizes: a few years ago Python's pickle code grew support for "extension codes", a registry of class/type names that _can_ be referenced by short (as short as 2 bytes) new pickle codes, instead of embedding the module and class name into every pickle, over and over again. I don't recall the exact numbers numbers, but some years ago Jeremy analyzed a customer Data.fs, and discovered that at least half of it consisted of repetitions of the string "BTrees.OOBTree.OOBTree" ;-) That's the kind of thing the "extension code" pickle mechanism was intended to address; it's a simple and cheap compression gimmick, but so far unused. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Re: [Zope] DateTime mess
[Chris Withers] > Sorry, my question was that if DateTime's were persistent, would the > following code result in a complete pickle for 'a' being written on the > second transaction commit? There is only one commit in the following, so I'll assume you intended a second commit at the end: > a.someTime = DateTime() > get_transaction().commit() Is `a` persistent? I'm assuming that it is. > ...wait/do stuff... > a.someTime = DateTime() I'm assuming another get_transaction().commit() was intended here. > ...have we just committed a pickle containing all of 'a'?... If `a` is persistent, yes. > Does mixing persistence into DateTime make a difference here? No. Assuming `a` is persistent, you changed `a`, so `a`'s state gets written out. This really has nothing to do with DateTime. It would be the same answer had you done, e.g., a.someTime = None or a.someTime = "hi!" or a.someTime = OOBTree() instead. One part is different: if DateTime is not persistent, then the pickle for a's state includes the full state of a.someTime's DateTime attribute (and this is true for any attribute of any non-persistent type). But if DateTime is persistent, then a.someTime is pickled as a distinct object, and the state for `a` contains only a reference to that distinct pickle (and again this is true for any attribute of any persistent type). a's full state is pickled regardless. The difference is in whether a's full state embeds a.someTime's state directly, or "points to" a distinct pickle for a.someTime's state. If the same DateTime object is referenced by many persistent objects, there can be major storage efficiencies is making DateTime persistent, because the "many other objects" can all point to the same pickle. I don't know, but I doubt that's a likely scenario for DateTime objects. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope] DateTime mess
On Dec 1, 2005, at 11:39 AM, Chris Withers wrote: Sorry, my question was that if DateTime's were persistent, would the following code result in a complete pickle for 'a' being written on the second transaction commit? a.someTime = DateTime() get_transaction().commit() wait/do stuff... a.someTime = DateTime() have we just committed a pickle containing all of 'a'?... Not quite. But if you call commit() again, you do indeed save a new pickle of a. Does mixing persistence into DateTime make a difference here? Not with respect to whether or not a second commit contains a pickle of a. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope] DateTime mess
On Dec 1, 2005, at 11:38 AM, Tim Peters wrote: ... I know Gary Poster gave a lot of thought to making pickles for the timezone info in Zope3 efficient too. For some definition of "a lot of thought". :-) The pickle for pytz.utc is now relatively small (though still adds a non-trivial percentage addition--30%ish?--to a naive datetime IIRC). That's as far as that bit goes. Gary ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: [Zope] DateTime mess
Tim Peters wrote: Sorry, I couldn't find a comprehensible question here after reasonable effort to extract one. Clearly, Zope2's DateTime.DateTime.DateTime objects are neither persistent nor do they define any mutating methods. Are those relevant? If not, try to ask a question directly, without assuming everyone has read the other 55 messages in the zope-dev thread ;-) Sorry, my question was that if DateTime's were persistent, would the following code result in a complete pickle for 'a' being written on the second transaction commit? a.someTime = DateTime() get_transaction().commit() ...wait/do stuff... a.someTime = DateTime() ...have we just committed a pickle containing all of 'a'?... Does mixing persistence into DateTime make a difference here? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Re: [Zope] DateTime mess
[Jürgen Herrmann] > the question was wether DateTime instances (of the new implementation, > which is yet to be coded) should mixin Persistent. OK. Since ZODB doesn't care whether you do, is there confusion about what ZODB may or may not do in either case? That is, what's the ZODB issue here? Have you looked at what Zope3 does? I believe it uses Python's new(ish) datetime.datetime objects as a base, and those aren't persistent either. In Zope2-land, a DateTime.DateTime is a large object, and you may have _wanted_ to make them persistent just so they could lose most of their memory-hogging state when ghostified. The state for a datetime.datetime object is much smaller, and I know Gary Poster gave a lot of thought to making pickles for the timezone info in Zope3 efficient too. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Re: [Zope] DateTime mess
the question was wether DateTime instances (of the new implementation, which is yet to be coded) should mixin Persistent. regards, juergen herrmann [ Tim Peters wrote:] > Sorry, I couldn't find a comprehensible question here after reasonable > effort to extract one. Clearly, Zope2's DateTime.DateTime.DateTime > objects > are neither persistent nor do they define any mutating methods. Are those > relevant? If not, try to ask a question directly, without assuming > everyone > has read the other 55 messages in the zope-dev thread ;-) > > -Original Message- > From: Chris Withers <[EMAIL PROTECTED]> > Sent: Wednesday, November 30, 2005 2:03 PM > To: Lennart Regebro > Cc: zodb-dev@zope.org > Subject: [ZODB-Dev] Re: [Zope] DateTime mess > > Lennart Regebro wrote: >> On 11/29/05, Chris Withers <[EMAIL PROTECTED]> wrote: >> >>>Hmm... how so? I've always thought it quite nice that when, for example, >>>you store the modification time of an object in a DateTime, you can >>>safely update it without worrying about the whole object having to be >>>repickled when you change it... >> >> Oh, so that make a difference? datetime objects are unchangeable, > > Really? didn't know that... > >> so >> everytime you change it you will have to set the attribute, and I >> thought the whole object got pickled then, but I guess that it >> doesn't, then. > > Honestly, I don't know. Changing to the zodb-dev list so someone who > does might comment... > > cheers, > > Chris > > -- > Simplistix - Content Management, Zope & Python Consulting > - http://www.simplistix.co.uk > > ___ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zodb-dev > ___ >> XLhost.de - eXperts in Linux hosting << Jürgen Herrmann Bruderwöhrdstraße 15b, DE-93051 Regensburg Fon: +49 (0)700 XLHOSTDE [0700 95467833] Fax: +49 (0)721 151 463027 WEB: http://www.XLhost.de ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] Re: [Zope] DateTime mess
Sorry, I couldn't find a comprehensible question here after reasonable effort to extract one. Clearly, Zope2's DateTime.DateTime.DateTime objects are neither persistent nor do they define any mutating methods. Are those relevant? If not, try to ask a question directly, without assuming everyone has read the other 55 messages in the zope-dev thread ;-) -Original Message- From: Chris Withers <[EMAIL PROTECTED]> Sent: Wednesday, November 30, 2005 2:03 PM To: Lennart Regebro Cc: zodb-dev@zope.org Subject: [ZODB-Dev] Re: [Zope] DateTime mess Lennart Regebro wrote: > On 11/29/05, Chris Withers <[EMAIL PROTECTED]> wrote: > >>Hmm... how so? I've always thought it quite nice that when, for example, >>you store the modification time of an object in a DateTime, you can >>safely update it without worrying about the whole object having to be >>repickled when you change it... > > Oh, so that make a difference? datetime objects are unchangeable, Really? didn't know that... > so > everytime you change it you will have to set the attribute, and I > thought the whole object got pickled then, but I guess that it > doesn't, then. Honestly, I don't know. Changing to the zodb-dev list so someone who does might comment... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
