Re: [Zope3-Users] Modified IntId utility
On Sep 25, 2007, at 8:52 AM, Stefan H. Holek wrote: On 24. Sep 2007, at 18:53, Maken Seteva wrote: Fellow Zopers. I have made a slight modification to my intid util by inheriting from IntId and overriding _generateId(). I do this to be sure that all new created objects will have an incrementing number where the first object created gets id 1, the next gets id 2 and so on. This way I get for free sorting by oldest/newest. Please do not attach additional meaning to intids. The value of an object's intid is (and should be) completely irrelevant. [snip] What do you think about this? Is it safe to do this, or have i forgotten any unforseen oddities that might occur in the future :O Your code looks like a conflict-magnet to me. I suggest you read up on why counting in the ZODB is a bad idea. Stefan -- It doesn't necessarily do it in chronological order, though. --Douglas Adams Hi Stefan, Looks like a few others besides me actually are giving meaning to intids as well.. I base my choice of adding meaning to ids from this post (one approach of reducing sort cost): Quote (by Jim Fulton http://www.mail-archive.com/[EMAIL PROTECTED]/ msg02144.html): Sort your results based on the primary key and therefore, pick your primary key to match your sort results. In terms of the Zope catalog framework, the primary keys are the document IDs, which are traditionally chosen randomly. You can pick your primary keys based on a desired sort order instead. A variation on this theme is to use multiple sets of document ids, storing multiple sets of ids in each index. Of course, this approach doesn't help with something like relevance ranks. End quote I *could* use, as the quote states, several ids but why not use the regular intid (that we already have) for this? As long as I resolve conflicts of course. Not fully sure how to resolve conflicts though as I'm quite new to this. I saw a suggestion on BTrees.Length in another thread. So could I easily just replace the nextid in my code to use a Length instead? (and use Length.change to increment the counter). My best regards Seteva___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Modified IntId utility
Maken Seteva wrote: On 24. Sep 2007, at 18:53, Maken Seteva wrote: Fellow Zopers. I have made a slight modification to my intid util by inheriting from IntId and overriding _generateId(). I do this to be sure that all new created objects will have an incrementing number where the first object created gets id 1, the next gets id 2 and so on. This way I get for free sorting by oldest/newest. Not fully sure how to resolve conflicts though as I'm quite new to this. The best way to resolve conflicts is to not create them. If you use the most significant bits of your int ID as a counter, and fill the least significant bits with conflict busting randomness, you'll be in good shape. We (ZC) have a package that does just that which we really need to release. If there's interest I'll take a stab at doing that soon. -- Benji York Senior Software Engineer Zope Corporation ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Modified IntId utility
On Sep 26, 2007, at 12:48 PM, Benji York wrote: The best way to resolve conflicts is to not create them. If you don't create conflicts, how can you resolv conflicts (that doesn't exist) ;-) If you use the most significant bits of your int ID as a counter, and fill the least significant bits with conflict busting randomness, you'll be in good shape. We (ZC) have a package that does just that which we really need to release. If there's interest I'll take a stab at doing that soon. -- Benji York Senior Software Engineer Zope Corporation Yes, I would be very interested in that package. Very much appreciated if you released it. Thanks for your help ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Modified IntId utility
Hi Benji, We (ZC) have a package that does just that which we really need to release. If there's interest I'll take a stab at doing that soon. Yes, I would be very interested in that package. Very much appreciated if you released it. Hey! I'm also very interested on this package. Could I help you with something to release it? Bye, Nando. -- http://www.nandoquintana.com/contact signature.asc Description: Esta parte del mensaje está firmada digitalmente ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Modified IntId utility
On Sep 26, 2007, at 12:48 PM, Benji York wrote: Maken Seteva wrote: On 24. Sep 2007, at 18:53, Maken Seteva wrote: Fellow Zopers. I have made a slight modification to my intid util by inheriting from IntId and overriding _generateId(). I do this to be sure that all new created objects will have an incrementing number where the first object created gets id 1, the next gets id 2 and so on. This way I get for free sorting by oldest/newest. Not fully sure how to resolve conflicts though as I'm quite new to this. The best way to resolve conflicts is to not create them. If you use the most significant bits of your int ID as a counter, and fill the least significant bits with conflict busting randomness, you'll be in good shape. We (ZC) have a package that does just that which we really need to release. If there's interest I'll take a stab at doing that soon. Can't wait so :) Otherwise I wrote a simple one in 10 lines of python: class Counter(object): def __init__(self, leastSignificantBits=4): self.incrementStep = 0x1 leastSignificantBits self.counter = 0 self.maxrand = pow(2, leastSignificantBits) def _increment(self): self.counter += self.incrementStep def __call__(self): self._increment() # OR the random bits at the end return self.counter | random.randrange(0, self.maxrand) Test: import myintid c = myintid.Counter(8) print hex(c()) 0x181 for _ in range(17): ... print hex(c()) ... 0x277 0x3aa 0x456 0x5ad 0x6bf 0x794 0x80f 0x9ef 0xade ... Maybe someone finds it useful /Seteva___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Modified IntId utility
On 24. Sep 2007, at 18:53, Maken Seteva wrote: Fellow Zopers. I have made a slight modification to my intid util by inheriting from IntId and overriding _generateId(). I do this to be sure that all new created objects will have an incrementing number where the first object created gets id 1, the next gets id 2 and so on. This way I get for free sorting by oldest/newest. Please do not attach additional meaning to intids. The value of an object's intid is (and should be) completely irrelevant. [snip] What do you think about this? Is it safe to do this, or have i forgotten any unforseen oddities that might occur in the future :O Your code looks like a conflict-magnet to me. I suggest you read up on why counting in the ZODB is a bad idea. Stefan -- It doesn't necessarily do it in chronological order, though. --Douglas Adams ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Modified IntId utility
Hi, On Mon, Sep 24, 2007 at 06:53:58PM +0200, Maken Seteva wrote: Fellow Zopers. I have made a slight modification to my intid util by inheriting from IntId and overriding _generateId(). I do this to be sure that all new created objects will have an incrementing number where the first object created gets id 1, the next gets id 2 and so on. This way I get for free sorting by oldest/newest. class MyIntIds(IntId): # We need a non-volatile nextid nextid = None def _generateId(self): # In the unlikely event that the nextid already # exists, we run this in a while loop to fix it. while True: if len(self) == 0: self.nextid = 1 elif self.nextid is None: self.nextid = self.refs.maxKey() + 1 uid = self.nextid self.nextid += 1 if uid not in self.refs: return uid # Normally we would never get here.. self.nextid = None What do you think about this? Is it safe to do this, or have i forgotten any unforseen oddities that might occur in the future :O I'm using a modified IntId utility which starts IDs at 1 and counts upwards, too. I needed some permanent link for each object which was easy to implement via IntId. The modification was made to safe keystrokes when typing the permalink. Typing http://server/~251 is a lot easier than http://server/~344357462 :-) . However, I don't use those IDs for anything else - like sorting. Regards, Frank ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] Modified IntId utility
On Sep 25, 2007, at 3:27 AM, Frank Burkhardt wrote: I'm using a modified IntId utility which starts IDs at 1 and counts upwards, too. I needed some permanent link for each object which was easy to implement via IntId. The modification was made to safe keystrokes when typing the permalink. Typing http://server/~251 is a lot easier than http://server/~344357462 :-) . However, I don't use those IDs for anything else - like sorting. We do. We use 64-bit intids arranged in ways that help with our most common sorting requirements (usually time related). Higher-order bits provide the sort order we care about, and the remaining lower- order bits provide randomness so that we reduce the chance of conflict errors. (It still can have a higher theoretical chance for conflict errors than the classic intid approach to assigning keys, but the BTree conflict resolution algorithm should eliminate the increase in practice.) So, we don't follow the exact pattern in the original post, and using the general pattern has subtleties you need to get right. For instance, you need to avoid conflict errors, as Stefan mentioned; and, for simplicity, we regard intids to be immutable and so they should only be tied to immutable data points about your objects. But we believe the idea--letting intids have meaning, particularly for sorting--has merit generally. ...sigh, we need to do some open-sourcing. We've been pretty busy. Gary ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
[Zope3-Users] Modified IntId utility
Fellow Zopers. I have made a slight modification to my intid util by inheriting from IntId and overriding _generateId(). I do this to be sure that all new created objects will have an incrementing number where the first object created gets id 1, the next gets id 2 and so on. This way I get for free sorting by oldest/newest. class MyIntIds(IntId): # We need a non-volatile nextid nextid = None def _generateId(self): # In the unlikely event that the nextid already # exists, we run this in a while loop to fix it. while True: if len(self) == 0: self.nextid = 1 elif self.nextid is None: self.nextid = self.refs.maxKey() + 1 uid = self.nextid self.nextid += 1 if uid not in self.refs: return uid # Normally we would never get here.. self.nextid = None What do you think about this? Is it safe to do this, or have i forgotten any unforseen oddities that might occur in the future :O Regards Seteva___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
