Re: [Zope3-dev] possible bug in catalog code

2005-07-08 Thread Dieter Maurer
Tim Peters wrote at 2005-7-7 15:03 -0400:
 ...
[Dieter]
 However, there is an additional reason not to put None into a BTree.

  Its interface has a weekness in its keys, values, min and max
  methods.

  E.g. in keys(min,max), min=None or max=None mean: no restriction.

  Therefore, they cannot distinguish between a None stored in the
  BTree itself and the use of None as no restriction.

[Tim]
Yup, except I wouldn't call it a weakness.  Since BTrees should not be
used with mixed key types anyway, and None is only value of type
type(None), the only BTrees having None as a key should be empty or
contain a single key (namely, None).  Range search on trees with 0 or
1 elements isn't particularly difficult wink.

Restrictions are fine when they are either

  *  enforced (I know BTrees do not enforce the no mixed key types
 restriction)

  *  clearly documented

 I do not know whether the ZODB 3.4 BTrees interfaces document
 the restriction (this would be a good place, I think)

 Hopefully, they do ;-)


-- 
Dieter
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com



Re: [Zope3-dev] possible bug in catalog code

2005-07-08 Thread Tim Peters
[Dieter Maurer]
 Restrictions are fine when they are either

  *  enforced (I know BTrees do not enforce the no mixed key types
 restriction)

  *  clearly documented

 I do not know whether the ZODB 3.4 BTrees interfaces document
 the restriction (this would be a good place, I think)

 Hopefully, they do ;-)

No, they don't.  Patches accepted 0.5 wink.  There are many ways to
get in trouble with BTrees (mixing key types is just one), and the
ZODB Programming Guide has pages of discussion of potential pitfalls
(including mixing key types) already:

http://www.zope.org/Wikis/ZODB/FrontPage/guide/node6.html

(and a copy of that in PDF format is shipped in every ZODB release).

I don't think mixing key types is, e.g., more important than the other
ways to create an insane BTree, and I don't think pages of discussion
fit sanely in interface docstrings either.  This is something that
belongs in documentation.  Since it already is covered in the only
official ZODB docs, I'm not inclined to do more than that.
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com



Re: [Zope3-dev] possible bug in catalog code

2005-07-07 Thread Jim Fulton

Martijn Faassen wrote:

Hi there,

in zope.app.catalog.attribute, there's the following functionality:

def index_doc(self, docid, object):
if self.interface is not None:
object = self.interface(object, None)
if object is None:
return None

value = getattr(object, self.field_name, None)
if value is None:
return None

if self.field_callable:
try:
value = value()
except:
return None

return super(AttributeIndex, self).index_doc(docid, value)

The following sequence I think leads to trouble:


value = getattr(object, self.field_name, None)
if value is None:
return None

as this means attributes that do exist and have the value None would 
never be indexed (as index_doc() is never reached).


Instead, I'd suggest the following code:

try:
value = getattr(object, self.field_name)
except AttributeError:
return None

or alternatively if for some reason getattr() with a default is 
preferred (though I suspect the exception approach is slightly faster):


# module level
NotFound = object()

value = getattr(object, self.field_name, NotFound)
if value is NotFound:
 return None

If this is indeed decided to be a bug, is it okay if I check in a fix 
for this? I'll try to devise a test of course.


I'm not sure if this is a bug.  If it is, the fix might be worse.
It makes no sense and could cause problems to use None as a BTree-bases
index key. AFAICT, Python still makes no guarantee wrt comparison of
of objects of different types over time:

  http://docs.python.org/ref/comparisons.html

  The operators , , ==, =, =, and != compare the values of two objects.
   The objects need not have the same type. If both are numbers, they are
   converted to a common type. Otherwise, objects of different types always
   compare unequal, and are ordered consistently but arbitrarily.

   (This unusual definition of comparison was used to simplify the definition of
   operations like sorting and the in and not in operators. In the future, the
   comparison rules for objects of different types are likely to change.)

It would only be safe to use None as a BTree key if all of the keys used
were None, which wouldn't be very interesting. :)

There are various ways that one could design an index that would
handle objects of different types, if necessary or handle None as
a special case.  Thar would require a fairly different design.

Jim

--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com



Re: [Zope3-dev] possible bug in catalog code

2005-07-07 Thread Martijn Faassen

Jim Fulton wrote:
[snip]

It would only be safe to use None as a BTree key if all of the keys used
were None, which wouldn't be very interesting. :)


It'd also make sense if you didn't do a range query, right? I.e. you're 
just looking for (None, None). I realize though that queries with None 
do something else entirely, as they open the range.


Anyway, perhaps a comment should be in place specifying that None cannot 
be indexed; this way it just looked surprising.


Now I'm going to figure out why an object that has a repr() with the 
same memory address in it isn't itself when the 'is' operator is used, 
and one of them doesn't have a __name__ and __parent__. This is the next 
thing that's confusing our cataloging.


Regards,

Martijn
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com



Re: [Zope3-dev] possible bug in catalog code

2005-07-07 Thread Jim Fulton

Martijn Faassen wrote:

Jim Fulton wrote:
[snip]


It would only be safe to use None as a BTree key if all of the keys used
were None, which wouldn't be very interesting. :)



It'd also make sense if you didn't do a range query, right? I.e. you're 
just looking for (None, None). I realize though that queries with None 
do something else entirely, as they open the range.


Any lookup in a BTree is a range query -- it depends on orderability
of the keys.

Anyway, perhaps a comment should be in place specifying that None cannot 
be indexed; this way it just looked surprising.


Sure.  There should probably be a note that all of the keys *must*
be orderable in a way that is consistent accross processes and Python
versons.  This is a far more likely gotcha.

Jim

--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com



Re: [Zope3-dev] possible bug in catalog code

2005-07-07 Thread Tim Peters
[Jim Fulton]
| ...
 AFAICT, Python still makes no guarantee wrt comparison of
 of objects of different types over time:

That's true, and in particular the relative (wrt objects of non-None
types) ordering of None _has_ changed across CPython releases.

The Total Ordering and Persistence section of the ZODB Programming
Guide explains some BTree requirements in detail, including why, e.g.,
it's not even safe to mix str keys with unicode keys:

http://www.zope.org/Wikis/ZODB/FrontPage/guide/node6.html
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com



Re: [Zope3-dev] possible bug in catalog code

2005-07-07 Thread Dieter Maurer
Jim Fulton wrote at 2005-7-7 06:55 -0400:
 ...
[Martijn]
 The following sequence I think leads to trouble:
 
 
 value = getattr(object, self.field_name, None)
 if value is None:
 return None
 
 as this means attributes that do exist and have the value None would 
 never be indexed (as index_doc() is never reached).
 ...
[Jim]

I'm not sure if this is a bug.  If it is, the fix might be worse.
It makes no sense and could cause problems to use None as a BTree-bases
index key. AFAICT, Python still makes no guarantee wrt comparison of
of objects of different types over time:

The current Python falls back to order by type name and than by id.
I do not expect this to change again.

However, there is an additional reason not to put None into a BTree.

  Its interface has a weekness in its keys, values, min and max
  methods.

  E.g. in keys(min,max), min=None or max=None mean: no restriction.

  Therefore, they cannot distinguish between a None stored in the
  BTree itself and the use of None as no restriction.


-- 
Dieter
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com



Re: [Zope3-dev] possible bug in catalog code

2005-07-07 Thread Tim Peters
[Dieter Maurer]
 The current Python falls back to order by type name and than by id.

Except that in current CPython releases, None is an exception to that, like

 class C(object):
... pass
 c = C()
 None  c  # None is less than c
True
 type(None).__name__  type(c).__name__ # although its type name is 
False

In some earlier CPython releases, None was not an exception to that.

 I do not expect this to change again.

I do, but perhaps not before Python 3000.  The trend among newer
builtin Python types is to allow mixed-type comparison _only_ when the
comparison operator is == or != (and objects of different types
compare not equal then).  For example, the various newer datetime.*
and set types work that way; e.g., try comparing a set to an integer
in Python 2.4:

 set() == 2
False
 set() != 2
True
 set()  2
Traceback (most recent call last):
  File stdin, line 1, in ?
TypeError: can only compare to a set

It's almost certainly the case, e.g., that trying to compare a str
with a float will eventually do the same kind of thing.

 However, there is an additional reason not to put None into a BTree.

  Its interface has a weekness in its keys, values, min and max
  methods.

  E.g. in keys(min,max), min=None or max=None mean: no restriction.

  Therefore, they cannot distinguish between a None stored in the
  BTree itself and the use of None as no restriction.

Yup, except I wouldn't call it a weakness.  Since BTrees should not be
used with mixed key types anyway, and None is only value of type
type(None), the only BTrees having None as a key should be empty or
contain a single key (namely, None).  Range search on trees with 0 or
1 elements isn't particularly difficult wink.
___
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com