Re: [Zope3-dev] possible bug in catalog code
[Dieter Maurer] > Restrictions are fine when they are either > > * enforced (I know BTrees do not enforce the "no mixed key types" > restriction) > > * clearly documented > > I do not know whether the ZODB 3.4 BTrees interfaces document > the restriction (this would be a good place, I think) > > Hopefully, they do ;-) No, they don't. Patches accepted <0.5 wink>. There are many ways to get in trouble with BTrees (mixing key types is just one), and the ZODB Programming Guide has pages of discussion of potential pitfalls (including mixing key types) already: http://www.zope.org/Wikis/ZODB/FrontPage/guide/node6.html (and a copy of that in PDF format is shipped in every ZODB release). I don't think mixing key types is, e.g., more important than the other ways to create an insane BTree, and I don't think pages of discussion fit sanely in interface docstrings either. This is something that belongs in documentation. Since it already is covered in the only "official" ZODB docs, I'm not inclined to do more than that. ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] possible bug in catalog code
Tim Peters wrote at 2005-7-7 15:03 -0400: > ... [Dieter] >> However, there is an additional reason not to put "None" into a BTree. >> >> Its interface has a weekness in its "keys", "values", "min" and "max" >> methods. >> >> E.g. in "keys(min,max)", "min=None" or "max=None" mean: no restriction. >> >> Therefore, they cannot distinguish between a "None" stored in the >> BTree itself and the use of "None" as "no restriction". [Tim] >Yup, except I wouldn't call it a weakness. Since BTrees should not be >used with mixed key types anyway, and None is only value of type >type(None), the only BTrees having None as a key should be empty or >contain a single key (namely, None). Range search on trees with 0 or >1 elements isn't particularly difficult . Restrictions are fine when they are either * enforced (I know BTrees do not enforce the "no mixed key types" restriction) * clearly documented I do not know whether the ZODB 3.4 BTrees interfaces document the restriction (this would be a good place, I think) Hopefully, they do ;-) -- Dieter ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] possible bug in catalog code
[Dieter Maurer] > The current Python falls back to order by type name and than by id. Except that in current CPython releases, None is an exception to that, like >>> class C(object): ... pass >>> c = C() >>> None < c # None is less than c True >>> type(None).__name__ < type(c).__name__ # although its type name is > False In some earlier CPython releases, None was not an exception to that. > I do not expect this to change again. I do, but perhaps not before "Python 3000". The trend among newer builtin Python types is to allow mixed-type comparison _only_ when the comparison operator is == or != (and objects of different types compare "not equal" then). For example, the various newer datetime.* and set types work that way; e.g., try comparing a set to an integer in Python 2.4: >>> set() == 2 False >>> set() != 2 True >>> set() < 2 Traceback (most recent call last): File "", line 1, in ? TypeError: can only compare to a set It's almost certainly the case, e.g., that trying to compare a str with a float will eventually do the same kind of thing. > However, there is an additional reason not to put "None" into a BTree. > > Its interface has a weekness in its "keys", "values", "min" and "max" > methods. > > E.g. in "keys(min,max)", "min=None" or "max=None" mean: no restriction. > > Therefore, they cannot distinguish between a "None" stored in the > BTree itself and the use of "None" as "no restriction". Yup, except I wouldn't call it a weakness. Since BTrees should not be used with mixed key types anyway, and None is only value of type type(None), the only BTrees having None as a key should be empty or contain a single key (namely, None). Range search on trees with 0 or 1 elements isn't particularly difficult . ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] possible bug in catalog code
Jim Fulton wrote at 2005-7-7 06:55 -0400: > ... [Martijn] >> The following sequence I think leads to trouble: >> >> >> value = getattr(object, self.field_name, None) >> if value is None: >> return None >> >> as this means attributes that do exist and have the value None would >> never be indexed (as index_doc() is never reached). > ... [Jim] >I'm not sure if this is a bug. If it is, the fix might be worse. >It makes no sense and could cause problems to use None as a BTree-bases >index key. AFAICT, Python still makes no guarantee wrt comparison of >of objects of different types over time: The current Python falls back to order by type name and than by id. I do not expect this to change again. However, there is an additional reason not to put "None" into a BTree. Its interface has a weekness in its "keys", "values", "min" and "max" methods. E.g. in "keys(min,max)", "min=None" or "max=None" mean: no restriction. Therefore, they cannot distinguish between a "None" stored in the BTree itself and the use of "None" as "no restriction". -- Dieter ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] possible bug in catalog code
[Jim Fulton] |> ... > AFAICT, Python still makes no guarantee wrt comparison of > of objects of different types over time: That's true, and in particular the relative (wrt objects of non-None types) ordering of None _has_ changed across CPython releases. The "Total Ordering and Persistence" section of the ZODB Programming Guide explains some BTree requirements in detail, including why, e.g., it's not even safe to mix str keys with unicode keys: http://www.zope.org/Wikis/ZODB/FrontPage/guide/node6.html ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] possible bug in catalog code
Martijn Faassen wrote: Jim Fulton wrote: [snip] It would only be safe to use None as a BTree key if all of the keys used were None, which wouldn't be very interesting. :) It'd also make sense if you didn't do a range query, right? I.e. you're just looking for (None, None). I realize though that queries with None do something else entirely, as they open the range. Any lookup in a BTree is a range query -- it depends on orderability of the keys. Anyway, perhaps a comment should be in place specifying that None cannot be indexed; this way it just looked surprising. Sure. There should probably be a note that all of the keys *must* be orderable in a way that is consistent accross processes and Python versons. This is a far more likely gotcha. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] possible bug in catalog code
Jim Fulton wrote: [snip] It would only be safe to use None as a BTree key if all of the keys used were None, which wouldn't be very interesting. :) It'd also make sense if you didn't do a range query, right? I.e. you're just looking for (None, None). I realize though that queries with None do something else entirely, as they open the range. Anyway, perhaps a comment should be in place specifying that None cannot be indexed; this way it just looked surprising. Now I'm going to figure out why an object that has a repr() with the same memory address in it isn't itself when the 'is' operator is used, and one of them doesn't have a __name__ and __parent__. This is the next thing that's confusing our cataloging. Regards, Martijn ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] possible bug in catalog code
Martijn Faassen wrote: Hi there, in zope.app.catalog.attribute, there's the following functionality: def index_doc(self, docid, object): if self.interface is not None: object = self.interface(object, None) if object is None: return None value = getattr(object, self.field_name, None) if value is None: return None if self.field_callable: try: value = value() except: return None return super(AttributeIndex, self).index_doc(docid, value) The following sequence I think leads to trouble: value = getattr(object, self.field_name, None) if value is None: return None as this means attributes that do exist and have the value None would never be indexed (as index_doc() is never reached). Instead, I'd suggest the following code: try: value = getattr(object, self.field_name) except AttributeError: return None or alternatively if for some reason getattr() with a default is preferred (though I suspect the exception approach is slightly faster): # module level NotFound = object() value = getattr(object, self.field_name, NotFound) if value is NotFound: return None If this is indeed decided to be a bug, is it okay if I check in a fix for this? I'll try to devise a test of course. I'm not sure if this is a bug. If it is, the fix might be worse. It makes no sense and could cause problems to use None as a BTree-bases index key. AFAICT, Python still makes no guarantee wrt comparison of of objects of different types over time: http://docs.python.org/ref/comparisons.html "The operators <, >, ==, >=, <=, and != compare the values of two objects. The objects need not have the same type. If both are numbers, they are converted to a common type. Otherwise, objects of different types always compare unequal, and are ordered consistently but arbitrarily. (This unusual definition of comparison was used to simplify the definition of operations like sorting and the in and not in operators. In the future, the comparison rules for objects of different types are likely to change.)" It would only be safe to use None as a BTree key if all of the keys used were None, which wouldn't be very interesting. :) There are various ways that one could design an index that would handle objects of different types, if necessary or handle None as a special case. Thar would require a fairly different design. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
