Re: [ZODB-Dev] create unique container keys
On Apr 2, 2009, at 10:22 AM, Alan Runyan wrote: > It seems like such an easy goal: autoincremental integers for a > container. That isn't the goal here. > Is this such a problem because of the ZODB architecture? It's such a problem because it isn't one problem (differing use cases) and because of scalability. > > or lack there of? There are two database primitives that everyone > appears to want: > > - autoincrementing integers for containers (tables) People want to generate keys. The keys they want can vary depending on requirements. ZODB already provides a mechanism for autogenerating unique ids. The storage API uses this for generating object ids. It's been proposed to add a general facility that could be used by applications. Generating ids sequentially across threads, even if you avoid id conflicts, is unattractive because you'll tend to provoke conflicts when BTrees split. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] create unique container keys
On Apr 2, 2009, at 4:19 AM, Adam GROSZER wrote: > Hello, > > What's a good practice to create unique container keys for a heavily > loaded application? (I mean lots of writes to the same container) > Obviously having a counter on the container and incrementing and using > that for key gives write conflicts. > > What do the experts use? Assuming that ordering of the keys isn't important, my favorite strategy is to: - use integer keys, which saves memory and makes lookup faster if you use an IOBTree. - assign keys sequentially with random starting points, using a _v_ variable to keep track of the index. The idea is that instances in separate threads/processes use different random starting points, to avoid conflicts, but assign keys sequentially to update the btree efficiently. See for example _generateId in: http://svn.zope.org/zope.intid/trunk/src/zope/intid/__init__.py?rev=95793&view=auto Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] create unique container keys
On Thu, 2009-04-02 at 16:41 +0200, Wichert Akkerman wrote: > On 4/2/09 4:36 PM, Christian Theune wrote: > > ZODB has autoincrement support for one type: OIDs. > > > > The problem of autoincrement is that this needs to be handled outside > > the scope of transactions. > > > > In a distributed fashion this seems rather hard to do in comparison to > > just buying into conflicts. > > > > I suspect it might be faster than the penalty of retrying a request on > conflict if used properly. Sure, hard problems might still have fast solutions. -- Christian Theune · c...@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] create unique container keys
On 4/2/09 4:36 PM, Christian Theune wrote: > ZODB has autoincrement support for one type: OIDs. > > The problem of autoincrement is that this needs to be handled outside > the scope of transactions. > > In a distributed fashion this seems rather hard to do in comparison to > just buying into conflicts. > I suspect it might be faster than the penalty of retrying a request on conflict if used properly. W. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] create unique container keys
On Thu, 2009-04-02 at 09:22 -0500, Alan Runyan wrote: > It seems like such an easy goal: autoincremental integers for a > container. Is this such a problem because of the ZODB architecture? > or lack there of? There are two database primitives that everyone > appears to want: > > - autoincrementing integers for containers (tables) > > - indexes (not in application) > > The new generation of databases (couchdb, tokyo cabinet, hypertable, > etc) autoincrementing is usually absent. Though these data containers > usually have index support. ZODB has autoincrement support for one type: OIDs. The problem of autoincrement is that this needs to be handled outside the scope of transactions. In a distributed fashion this seems rather hard to do in comparison to just buying into conflicts. The indexes are hard too. We have gocept.objectquery which is (still) in a proof-of-concept state, although I'm getting really motivated WRT using it for migrations. Christian -- Christian Theune · c...@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] create unique container keys
It seems like such an easy goal: autoincremental integers for a container. Is this such a problem because of the ZODB architecture? or lack there of? There are two database primitives that everyone appears to want: - autoincrementing integers for containers (tables) - indexes (not in application) The new generation of databases (couchdb, tokyo cabinet, hypertable, etc) autoincrementing is usually absent. Though these data containers usually have index support. cheers alan On Thu, Apr 2, 2009 at 7:48 AM, Stephan Richter wrote: > On Thursday 02 April 2009, Adam GROSZER wrote: >> Yeah, uuid was my guess. > > Christian was not necessarily thinking uuid. I think the common algorithm is: > > x = random.randint(10**9) > while str(x) in container: > x += 1 > > Regards, > Stephan > -- > Stephan Richter > Web Software Design, Development and Training > Google me. "Zope Stephan Richter" > ___ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - zodb-...@zope.org > http://mail.zope.org/mailman/listinfo/zodb-dev > -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] create unique container keys
On Thursday 02 April 2009, Adam GROSZER wrote: > Yeah, uuid was my guess. Christian was not necessarily thinking uuid. I think the common algorithm is: x = random.randint(10**9) while str(x) in container: x += 1 Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter" ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] create unique container keys
Hello Christian, Yeah, uuid was my guess. What about some human-eatable counter-like thing? Btrees have some length with conflict resolution support, but I'm not sure that could be used to make a key and __name__. Thursday, April 2, 2009, 10:47:07 AM, you wrote: CT> On Thu, 2009-04-02 at 10:19 +0200, Adam GROSZER wrote: >> Hello, >> >> What's a good practice to create unique container keys for a heavily >> loaded application? (I mean lots of writes to the same container) >> Obviously having a counter on the container and incrementing and using >> that for key gives write conflicts. >> >> What do the experts use? CT> If uniqueness is your only concern then using a large number range and CT> consuming numbers randomly will perform better than incrementing. CT> Christian -- Best regards, Adam GROSZERmailto:agros...@gmail.com -- Quote of the day: The golden opportunity you are seeking is in yourself. It is not in your environment; it is not in luck or chance, or the help of other; it is in yourself alone. (Orison Swett Marden) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] create unique container keys
On Thu, 2009-04-02 at 10:19 +0200, Adam GROSZER wrote: > Hello, > > What's a good practice to create unique container keys for a heavily > loaded application? (I mean lots of writes to the same container) > Obviously having a counter on the container and incrementing and using > that for key gives write conflicts. > > What do the experts use? If uniqueness is your only concern then using a large number range and consuming numbers randomly will perform better than incrementing. Christian -- Christian Theune · c...@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development signature.asc Description: This is a digitally signed message part ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] create unique container keys
Hello, What's a good practice to create unique container keys for a heavily loaded application? (I mean lots of writes to the same container) Obviously having a counter on the container and incrementing and using that for key gives write conflicts. What do the experts use? thanks -- Best regards, Adam GROSZER mailto:agros...@gmail.com -- Quote of the day: And the number one thing you'll never hear in an internet chat room: 1. Where have I been lately? Hey, some of us have a life, ya know! ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
