Re: [Python-Dev] Dict suppressing exceptions
M.-A. Lemburg wrote: M.-A. Lemburg wrote: Guido van Rossum wrote: Marc-Andre, how's the patch coming along? I'm working on it. Since we only want equal compares to generate the warning, I have to add a rich compare function to Unicode objects. Here's an initial version: http://sourceforge.net/tracker/index.php?func=detailaid=1538956group_id=5470atid=305470 The title of the patch is slightly incorrect - SF doesn't allow more descriptive titles... :-( FYI: I've checked in the patch as revision 51276. Please give it some serious testing. All tests pass, but there may still be some corner cases left which the tests don't catch. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 14 2006) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Hi, On Thu, Aug 10, 2006 at 02:36:16PM -0700, Guido van Rossum wrote: On Thu, Aug 10, 2006 at 09:11:42PM +0200, Martin v. L?wis wrote: I'm in favour of having this __eq__ just return False. I don't think the warning is necessary, (...) +1 Can you explain why you believe that no warning is necessary? Ah... mostly out of ignorance, I fear. I did not realize there were strange cases like u\xff == \xff. I will leave this to more unicode-minded people to decide, but I'm still definitely of the idea that the __eq__ should not raise an exception. A bientot, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Armin Rigo wrote: Hi, On Thu, Aug 10, 2006 at 02:36:16PM -0700, Guido van Rossum wrote: On Thu, Aug 10, 2006 at 09:11:42PM +0200, Martin v. L?wis wrote: I'm in favour of having this __eq__ just return False. I don't think the warning is necessary, (...) +1 Can you explain why you believe that no warning is necessary? Ah... mostly out of ignorance, I fear. I did not realize there were strange cases like u\xff == \xff. This is not all that strange. Perhaps this example looks closer to life ;-) if u'D\xfcsseldorf' != 'D\xfcsseldorf': move_to_Cologne() Now, we wouldn't want that to go through without a warning, do we... ;-) [Background: there's a bit of rivalry between those two cities; see e.g. see http://en.wikipedia.org/wiki/D%C3%BCsseldorf] -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 11 2006) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Guido writes: Alas, I have no idea what it does. Can you come up with an example that doesn't require enums and localization? Sorry. Here's the short version: Fact 1: Sometimes people create objects that raise exceptions when compared for equality. Maybe it's a bad idea to do this, and objects should never raise exceptions when compared except to signal internal inconsistancy -- but the practice is common enough to show up in a 5-star recipe for enums in the Cookbook. Fact 2: Unrelated objects are often put in dictionarys together. I used a localization dictionary as an example. I feel this is a legitimate practice. If dictionarys stop suppressing exceptions raised by equality tests then those two facts aren't compatible. Programs that used to work fine will break with 2.4. I'm OK with your proposed solution (fix the case of str-unicode but let everyone else using objects that raise exceptions suffer), but I think it would be more friendly to use warnings for now and exceptions after another full release cycle. The warnings should solve the underlying issue (hard-to-debug problems). -- Michael Chermside ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/11/06, Armin Rigo [EMAIL PROTECTED] wrote: Hi, On Thu, Aug 10, 2006 at 02:36:16PM -0700, Guido van Rossum wrote: On Thu, Aug 10, 2006 at 09:11:42PM +0200, Martin v. L?wis wrote: I'm in favour of having this __eq__ just return False. I don't think the warning is necessary, (...) +1 Can you explain why you believe that no warning is necessary? Ah... mostly out of ignorance, I fear. I did not realize there were strange cases like u\xff == \xff. I will leave this to more unicode-minded people to decide, but I'm still definitely of the idea that the __eq__ should not raise an exception. Me too, and that's what we'll do in py3k. But in 2.5, we're bound by the decisions we made in 1999-2000 about unicode. (Unless Martin has a convincing reason not to have a warning?) Marc-Andre, how's the patch coming along? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Guido van Rossum wrote: Marc-Andre, how's the patch coming along? I'm working on it. Since we only want equal compares to generate the warning, I have to add a rich compare function to Unicode objects. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 11 2006) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Guido van Rossum schrieb: Me too, and that's what we'll do in py3k. But in 2.5, we're bound by the decisions we made in 1999-2000 about unicode. (Unless Martin has a convincing reason not to have a warning?) Only the general anti-warning argument: it's not the developer which will see the warning, but the end user. In this case, they might easily get hundreds of warning in a short time, and these fill their Apache log files. They complain about Python eating their disk space. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Michael Chermside schrieb: I don't *strongly* object to this consensus, but if you haven't glanced at my original example, take a look - it might convince you. The proposed solution will not help with my example. I ignored your example the first time because it was too complicated to understand. Now I looked at it, and think that the recipe is broken. It should add an __eq__ method which is def __eq__(self, other): return type(self) is type(other) \ and self.EnumType is other.EnumType \ and self.__value==other.__value) Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Dict suppressing exceptions
Martin v. Löwis wrote: Now I looked at it, and think that the recipe is broken. It should add an __eq__ method which is def __eq__(self, other): return type(self) is type(other) \ and self.EnumType is other.EnumType \ and self.__value==other.__value) Absolutely. But the point is that there are a lot of these broken objects out there; this one was on a 5-star recipe that has been around for a while, and still no one caught it. That suggests the bug could reasonably be in any 3rd-party library. The existence of a (common) bug in someone else's code shouldn't keep me from using a dictionary of objects. hash was just changed to support the common use of id, even though some people argued it was *really* a bug in the classes themselves. This is a similar situation. -jJ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Martin v. Löwis writes: Now I looked at it, and think that the recipe is broken. Not broken, but perhaps wrongheaded. The recipe went out of its way to ensure that it would raise an exception of enum values from different enumerations were compared. There's nothing out there saying that this is a bad thing to do. I propose that we institute a new policy. The policy should state: __eq__ methods should always return True or False. They should only raise an exception if there is some internal error within one of the objects being compared -- they should never raise an exception because the other object is of an unexpected type. On the other hand, __lt__, __gt__ and friends SHOULD raise an exception when the object being compared is of some type for which the ordering does not make sense (e.g.: unicode vs byte-string or complex vs anything). I think we should note this policy someplace official -- perhaps in the Language Reference where __eq__ and __lt__ are defined. But I do not think that these changes should be made until Py3K. What do others think? Is this the right approach? -- Michael Chermside ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
M.-A. Lemburg wrote: Guido van Rossum wrote: Marc-Andre, how's the patch coming along? I'm working on it. Since we only want equal compares to generate the warning, I have to add a rich compare function to Unicode objects. Here's an initial version: http://sourceforge.net/tracker/index.php?func=detailaid=1538956group_id=5470atid=305470 The title of the patch is slightly incorrect - SF doesn't allow more descriptive titles... :-( -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 11 2006) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/11/06, Michael Chermside [EMAIL PROTECTED] wrote: Martin v. Löwis writes: Now I looked at it, and think that the recipe is broken. Not broken, but perhaps wrongheaded. The recipe went out of its way to ensure that it would raise an exception of enum values from different enumerations were compared. There's nothing out there saying that this is a bad thing to do. Except a little voice in my head. :-) I propose that we institute a new policy. The policy should state: __eq__ methods should always return True or False. They should only raise an exception if there is some internal error within one of the objects being compared -- they should never raise an exception because the other object is of an unexpected type. That's pretty much what I want to do in py3k. There, ordering will raise a TypeError by default, but == compares object identity by default, and it's strongly recommended to return False for any unrecognized type. I want the recommendation to be a *little* more relaxed than your proposal; IMO it's fine to raise an exception when your __eq__ implementation is incomplete, i.e. you haven't thought enough about how your class should compare to other classes. The don't silence exceptions rule will then cause loud complaints when your class gets compared to something it didn't expect being compared to, signalling that you have more work to do. (And yes, this can cause sleeper bombs when you use these objects as dict keys together with types that it doesn't understand, and the bomb will go off if there's a hash collision; but I've got fairly controlled situations in mind, not robust libraries or frameworks.) On the other hand, __lt__, __gt__ and friends SHOULD raise an exception when the object being compared is of some type for which the ordering does not make sense (e.g.: unicode vs byte-string or complex vs anything). Yes, that's the py3k rule. I think we should note this policy someplace official -- perhaps in the Language Reference where __eq__ and __lt__ are defined. But I do not think that these changes should be made until Py3K. Fine to update the docs now. What do others think? Is this the right approach? Yes. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/11/06, Martin v. Löwis [EMAIL PROTECTED] wrote: Michael Chermside schrieb: I propose that we institute a new policy. The policy should state: __eq__ methods should always return True or False. They should only raise an exception if there is some internal error within one of the objects being compared -- they should never raise an exception because the other object is of an unexpected type. That policy is currently difficult to implement, but reasonable (difficult because it is quite some code to write). Why? Are you thinking of the standard library, or of an end user's __eq__ method? Returning False from your __eq__ if other's type is unexpected doesn't seem a lot of code. Or am I misunderstanding something? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Guido van Rossum schrieb: __eq__ methods should always return True or False. They should only raise an exception if there is some internal error within one of the objects being compared -- they should never raise an exception because the other object is of an unexpected type. That policy is currently difficult to implement, but reasonable (difficult because it is quite some code to write). Why? Are you thinking of the standard library, or of an end user's __eq__ method? Returning False from your __eq__ if other's type is unexpected doesn't seem a lot of code. Or am I misunderstanding something? It seemed like a lot of code to me: In the specific enum example, I first wrote def __eq__(self, other): return self.EnumType is other.EnumType \ and self.__value==other.__value So this was wrong, as it did not take into account 'other' being something completely different, and I wrote def __eq__(self, other): return type(self) is type(other) \ and self.EnumType is other.EnumType \ and self.__value==other.__value Having two additional continuation lines seems quite difficult to me, yet a version that breaks the expression into multiple statements is even longer def __eq__(self, other): if type(self) is not type(other): return False if self.EnumType is not other.EnumType: return False return self.__value==other.__value Compare this to the current two-line __cmp__ implementation: def __cmp__(self, other): assert self.Enumtype is other.EnumType return cmp(self.__value, other.__value) This covers all comparisons just fine in two lines of method body; to implement the __eq__ policy, you need another 6 lines. For consistency, you should also implement __ne__, probably as def __ne__(self, other): return not self.__eq__(other) I expect many people get this wrong, for example http://pyref.infogami.com/__eq__ UserList.py Also, there is confusion as to whether NotImplemented should ever be returned in these. decimal.py believes it does (for different types), sets.py believes it doesn't. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Jim Jewett [EMAIL PROTECTED] writes: It wasn't my idea to stop ignoring exceptions in dict lookups; I would gladly have put this off until Py3k, where the main problem (str-unicode __eq__ raising UnicodeError) will go away. But since people are adamant that they want this in sooner, Is this true for dictionaries specifically? Would there really be strong objections to continuing to swallow any Exception (not BaseException) raised by __eq__ ? Armin's reason for changing dictionaries in this way was that enormous debugging pain was caused by dicts swallowing exceptions raised by __eq__ methods. Having the __eq__ methods swallow the exceptions by themselves would make the situation *worse*, not better. Cheers, mwh -- * vegai wears his reading bra. vegai umm, I mean glasses -- from Twisted.Quotes ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Guido van Rossum wrote:
I've been happily ignoring python-dev for the last three weeks or so,
and Neal just pointed me to some thorny issues that are close to
resolution but not quite yet resolved, yet need to be before beta 3 on
August 18 (Friday next week).
Here's my take on the dict-suppressing-exceptions issue (I'll send out
separate messages for each issue where Neal has asked me to weigh in).
It wasn't my idea to stop ignoring exceptions in dict lookups; I would
gladly have put this off until Py3k, where the main problem
(str-unicode __eq__ raising UnicodeError) will go away.
But since people are adamant that they want this in sooner, I suggest
that to minimize breakage we could make an exception for str-unicode
comparisons.
I came up with the following code to reproduce the issue; this prints
0 in 2.2, False in 2.3 and 2.4, but raises UnicodeDecodeError in 2.5
(head):
a = {u1: 1}
x = hash(u1)
class C(str):
def __hash__(s): return x
print C(\xff) in a
The following patch makes this print False in 2.5 again.
Notes about the patch:
- this also fixes an out-of-date comment that should be fixed even if
the rest of the idea is rejected (lookdict_string() can return NULL
when it calls lookdict)
- the exception could be narrowed even further by only suppressing the
exception when startkey and key are both either str or unicode
instances.
What do people think?
I'd suggest that we still inform the programmers of the problem
by issuing a warning (which they can then silence at will),
maybe a new PyExc_UnicodeWarning.
Note that these exceptions help programmers making their applications
Unicode compatible, so silencing them completely would remove the
possibility to detect the case of mixing strings and Unicode as
keys in a dictionary.
BTW, in Py3k, this case would not trigger at all, since all text
would be Unicode and bytes wouldn't be comparable to Unicode
anyway. However, that's a different discussion which we can have
after Python 2.5 is out the door.
--- Objects/dictobject.c (revision 51180)
+++ Objects/dictobject.c (working copy)
@@ -230,7 +230,8 @@
lookdict() is general-purpose, and may return NULL if (and only if) a
comparison raises an exception (this was new in Python 2.5).
lookdict_string() below is specialized to string keys, comparison of which
can
-never raise an exception; that function can never return NULL. For both,
when
+never raise an exception; that function can never return NULL (except when it
+decides to replace itself with the more general lookdict()). For both, when
the key isn't found a dictentry* is returned for which the me_value field is
NULL; this is the slot in the dict at which the key would have been found,
and
the caller can (if it wishes) add the key, value pair to the returned
@@ -259,8 +260,13 @@
if (ep-me_hash == hash) {
startkey = ep-me_key;
cmp = PyObject_RichCompareBool(startkey, key, Py_EQ);
- if (cmp 0)
- return NULL;
+ if (cmp 0) {
+ if (PyErr_Occurred()==PyExc_UnicodeDecodeError)
{
if (PyErr_Warn(PyExc_UnicodeWarning,
mixing unicode and strings
as dictionary keys) 0)
return NULL;
+PyErr_Clear();
+ }
+ else
+ return NULL;
+}
if (ep0 == mp-ma_table ep-me_key == startkey) {
if (cmp 0)
return ep;
@@ -289,8 +295,13 @@
if (ep-me_hash == hash ep-me_key != dummy) {
startkey = ep-me_key;
cmp = PyObject_RichCompareBool(startkey, key, Py_EQ);
- if (cmp 0)
- return NULL;
+ if (cmp 0) {
+ if (PyErr_Occurred()==PyExc_UnicodeDecodeError)
{
+ PyErr_Clear();
+ }
+ else
+ return NULL;
+}
if (ep0 == mp-ma_table ep-me_key == startkey) {
if (cmp 0)
return ep;
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Aug 10 2006)
Python/Zope Consulting and Support ...http://www.egenix.com/
mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/
Re: [Python-Dev] Dict suppressing exceptions
Guido van Rossum wrote: - the exception could be narrowed even further by only suppressing the exception when startkey and key are both either str or unicode instances. I always assumed dictionaries would work exactly like this. So, at least it would now work as I had always expected (and others possibly too). - Ralf ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/10/06, M.-A. Lemburg [EMAIL PROTECTED] wrote:
Guido van Rossum wrote:
But since people are adamant that they want this in sooner, I suggest
that to minimize breakage we could make an exception for str-unicode
comparisons.
What do people think?
I'd suggest that we still inform the programmers of the problem
by issuing a warning (which they can then silence at will),
maybe a new PyExc_UnicodeWarning.
BTW, in Py3k, this case would not trigger at all, since all text
would be Unicode and bytes wouldn't be comparable to Unicode
anyway. However, that's a different discussion which we can have
after Python 2.5 is out the door.
I strongly believe that unicode vs str here is the symptom and not the
actual problem. The comparison between two non-us-ascii str/unicode
instances is but one of many ways to raise an exception during
comparison. It's not even obvious ahead of time when it will occur.
Try my example below with (sys.maxint 1) and -2 instead of 1 and 1.
Do you expect a problem?
Because str/unicode is the common case, we saw it first. If we address
the symptom instead of the problem, someone will be complaining within
a years time because they have a class that they mix in with other
items for a function handler lookup, or who knows what, that works
like the following:
class hasher(object):
... def __init__(self, hashval):
... self.hashval = hashval
... def __hash__(self):
... return hash(self.hashval)
... def __eq__(self, o):
... if not isinstance(o, hasher):
... raise TypeError(Cannot compare hashval to non hashval)
... return self.hashval == o.hashval
in python2.4:
dict.fromkeys([1, hasher(1)])
{1: None, __main__.hasher object at 0xa7a5326c: None}
in python2.5b2:
dict.fromkeys([1, hasher(1)])
Traceback (most recent call last):
File stdin, line 1, in module
File stdin, line 8, in __eq__
TypeError: Cannot compare hashval to non hashval
Yes this is made up code. But I'm not arguing for a feature; I'm
arguing for backwards compatibility. Because we do not know where
these legitimate uses are, we cannot evaluate their likelihood to
exist nor the level of breakage they will cause. If we make this a
warning for any exception, we can satisfy both imagined camps. Those
in Armin's position can make that warning raise an exception while
debugging, and those using it on purpose can squash it.
I understand the utility of being able to see this case happening. I'm
not sure it's worth the warning. I'm positive it's not worth the
backwards incompatibility of the raised exception.
Michael
--
Michael Urman http://www.tortall.net/mu/blog
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Guido van Rossum wrote: On 8/10/06, M.-A. Lemburg [EMAIL PROTECTED] wrote: I'd suggest that we still inform the programmers of the problem by issuing a warning (which they can then silence at will), maybe a new PyExc_UnicodeWarning. Hmm... Here's an idea... How about we change unicode-vs-str __eq__ to issue a warning (and return False) instead of raising UnicodeException? That won't break much code (it's unlikely that people *depend* on this exception since it's generally a symptom of insane mixing of str and unicode). Then no further changes to dictobject.c are necessary (except fixing that one misleading comment). Good idea. Note that these exceptions help programmers making their applications Unicode compatible, so silencing them completely would remove the possibility to detect the case of mixing strings and Unicode as keys in a dictionary. A warning would arguably have the same helping effect. (I suspect actually that we would have used warnings all along except we didn't have the warning framework when unicode was first introduced.) Probably, yes. These days, I believe that we should have not gone for the string-Unicode integration magic at all and instead done the same of what's planned for Py3k. Sometimes a clean cut is better than trying to muddle along. BTW, in Py3k, this case would not trigger at all, since all text would be Unicode and bytes wouldn't be comparable to Unicode anyway. However, that's a different discussion which we can have after Python 2.5 is out the door. Which is why I said I would have gladly waited until Py3k to fix this. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 10 2006) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/10/06, M.-A. Lemburg [EMAIL PROTECTED] wrote: Guido van Rossum wrote: Hmm... Here's an idea... How about we change unicode-vs-str __eq__ to issue a warning (and return False) instead of raising UnicodeException? That won't break much code (it's unlikely that people *depend* on this exception since it's generally a symptom of insane mixing of str and unicode). Then no further changes to dictobject.c are necessary (except fixing that one misleading comment). Good idea. Great! Now we need someone to volunteer to write a patch (which should include doc and NEWS updates) in time for beta 3 (Aug 18). -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Michael Urman writes: I strongly believe that unicode vs str here is the symptom and not the actual problem. The comparison between two non-us-ascii str/unicode instances is but one of many ways to raise an exception during comparison. [... example ...] Yes this is made up code. But I'm not arguing for a feature; I'm arguing for backwards compatibility. Because we do not know where these legitimate uses are, we cannot evaluate their likelihood to exist nor the level of breakage they will cause. If we make this a warning for any exception, we can satisfy both imagined camps. Those in Armin's position can make that warning raise an exception while debugging, and those using it on purpose can squash it. See also my example from the beginning of this thread (http://mail.python.org/pipermail/python-dev/2006-August/067978.html). The example wasn't from real code, but it WAS quite plausible -- straightforward use of a popular Python Cookbook recipe. But we seem to have reached a rough consensus: Later, Guido writes: How about we change unicode-vs-str __eq__ to issue a warning (and return False) instead of raising UnicodeException? [... Marc-Andre Lemburg agrees ...] Great! Now we need someone to volunteer to write a patch (which should include doc and NEWS updates) in time for beta 3 (Aug 18). I don't *strongly* object to this consensus, but if you haven't glanced at my original example, take a look - it might convince you. The proposed solution will not help with my example. I'm not sure the motivation for breaking code like that example -- the bug-hunting motivation is satisfied by issuing a warning as Michael Urman proposes, then use an exception after one more release when people have had time to fix their code. -- Michael Chermside ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Hi, On Thu, Aug 10, 2006 at 09:11:42PM +0200, Martin v. L?wis wrote: I'm in favour of having this __eq__ just return False. I don't think the warning is necessary, (...) +1 Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/10/06, Michael Chermside [EMAIL PROTECTED] wrote: See also my example from the beginning of this thread (http://mail.python.org/pipermail/python-dev/2006-August/067978.html). The example wasn't from real code, but it WAS quite plausible -- straightforward use of a popular Python Cookbook recipe. [...] I don't *strongly* object to this consensus, but if you haven't glanced at my original example, take a look - it might convince you. Alas, I have no idea what it does. Can you come up with an example that doesn't require enums and localization? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Michael Chermside wrote: How about we change unicode-vs-str __eq__ to issue a warning (and return False) instead of raising UnicodeException? [... Marc-Andre Lemburg agrees ...] Great! Now we need someone to volunteer to write a patch (which should include doc and NEWS updates) in time for beta 3 (Aug 18). I don't *strongly* object to this consensus, but if you haven't glanced at my original example, take a look - it might convince you. The proposed solution will not help with my example. I'm not sure the motivation for breaking code like that example -- the bug-hunting motivation is satisfied by issuing a warning as Michael Urman proposes, then use an exception after one more release when people have had time to fix their code. The warning framework gives programmers great flexibility in handling these situation: * they can silence the warning, just report it once, always report it * they can have the warning raise an exception * they can log the warning in some way It's very flexible to accommodate for all kinds of ways to handle the situation. By introducing a new category of warning for these Unicode-related warnings, adding specific warning filters will be easy. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 10 2006) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Guido van Rossum wrote: On 8/10/06, M.-A. Lemburg [EMAIL PROTECTED] wrote: Guido van Rossum wrote: Hmm... Here's an idea... How about we change unicode-vs-str __eq__ to issue a warning (and return False) instead of raising UnicodeException? That won't break much code (it's unlikely that people *depend* on this exception since it's generally a symptom of insane mixing of str and unicode). Then no further changes to dictobject.c are necessary (except fixing that one misleading comment). Good idea. Great! Now we need someone to volunteer to write a patch (which should include doc and NEWS updates) in time for beta 3 (Aug 18). I'll give it a go. Not sure what kinds of docs we'll need for this, though (and more importantly: where to put them, e.g. into the tutorial in the dictionary section, somewhere into the Unicode section ?) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 10 2006) Python/Zope Consulting and Support ...http://www.egenix.com/ mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/10/06, Martin v. Löwis [EMAIL PROTECTED] wrote: Guido van Rossum schrieb: Hmm... Here's an idea... How about we change unicode-vs-str __eq__ to issue a warning (and return False) instead of raising UnicodeException? I'm in favour of having this __eq__ just return False. I don't think the warning is necessary, and believe that people will complain about getting it, but if that is the only way to achieve consensus, so be it. Not sure I agree. I think that users would find it disturbing (and it could hide bugs) if u\xff == \xff returned False without warning while u\x7f == \x7f returns True. A warning would arguably have the same helping effect. (I suspect actually that we would have used warnings all along except we didn't have the warning framework when unicode was first introduced.) I believe we would have neither had we rich comparisons available at the time. That unicode-vs-str-__eq__ raises an exception is just a fall-out of not supporting __cmp__ (and it is good that __cmp__ is not supported - there is no meaningful way to order str/unicode if the string is not in the system encoding). Maybe. I agree that 3-way compare has no business guessing the ordering of u\xff and \xff. I'm not so sure that if we'd had rich comparison we'd have chosen __eq__ to return False for these. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/10/06, Armin Rigo [EMAIL PROTECTED] wrote: Hi, On Thu, Aug 10, 2006 at 09:11:42PM +0200, Martin v. L?wis wrote: I'm in favour of having this __eq__ just return False. I don't think the warning is necessary, (...) +1 Can you explain why you believe that no warning is necessary? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
M.-A. Lemburg wrote: Michael Chermside wrote: How about we change unicode-vs-str __eq__ to issue a warning (and return False) instead of raising UnicodeException? [... Marc-Andre Lemburg agrees ...] Great! Now we need someone to volunteer to write a patch (which should include doc and NEWS updates) in time for beta 3 (Aug 18). I don't *strongly* object to this consensus, but if you haven't glanced at my original example, take a look - it might convince you. The proposed solution will not help with my example. I'm not sure the motivation for breaking code like that example -- the bug-hunting motivation is satisfied by issuing a warning as Michael Urman proposes, then use an exception after one more release when people have had time to fix their code. The warning framework gives programmers great flexibility in handling these situation: * they can silence the warning, just report it once, always report it * they can have the warning raise an exception * they can log the warning in some way It's very flexible to accommodate for all kinds of ways to handle the situation. By introducing a new category of warning for these Unicode-related warnings, adding specific warning filters will be easy. This sounds like the correct tool to me. +1 Would it be possible to generate warnings when either Unicode or stings are coerced to the other implicitly but not explicitly? That may also generate a warning in the case of the dictionary problem being discussed, (I think), and may be more useful for checking for non-intentionally mixed Unicode and strings. Ron ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Dict suppressing exceptions
I've been happily ignoring python-dev for the last three weeks or so,
and Neal just pointed me to some thorny issues that are close to
resolution but not quite yet resolved, yet need to be before beta 3 on
August 18 (Friday next week).
Here's my take on the dict-suppressing-exceptions issue (I'll send out
separate messages for each issue where Neal has asked me to weigh in).
It wasn't my idea to stop ignoring exceptions in dict lookups; I would
gladly have put this off until Py3k, where the main problem
(str-unicode __eq__ raising UnicodeError) will go away.
But since people are adamant that they want this in sooner, I suggest
that to minimize breakage we could make an exception for str-unicode
comparisons.
I came up with the following code to reproduce the issue; this prints
0 in 2.2, False in 2.3 and 2.4, but raises UnicodeDecodeError in 2.5
(head):
a = {u1: 1}
x = hash(u1)
class C(str):
def __hash__(s): return x
print C(\xff) in a
The following patch makes this print False in 2.5 again.
Notes about the patch:
- this also fixes an out-of-date comment that should be fixed even if
the rest of the idea is rejected (lookdict_string() can return NULL
when it calls lookdict)
- the exception could be narrowed even further by only suppressing the
exception when startkey and key are both either str or unicode
instances.
What do people think?
--- Objects/dictobject.c(revision 51180)
+++ Objects/dictobject.c(working copy)
@@ -230,7 +230,8 @@
lookdict() is general-purpose, and may return NULL if (and only if) a
comparison raises an exception (this was new in Python 2.5).
lookdict_string() below is specialized to string keys, comparison of which can
-never raise an exception; that function can never return NULL. For both, when
+never raise an exception; that function can never return NULL (except when it
+decides to replace itself with the more general lookdict()). For both, when
the key isn't found a dictentry* is returned for which the me_value field is
NULL; this is the slot in the dict at which the key would have been found, and
the caller can (if it wishes) add the key, value pair to the returned
@@ -259,8 +260,13 @@
if (ep-me_hash == hash) {
startkey = ep-me_key;
cmp = PyObject_RichCompareBool(startkey, key, Py_EQ);
- if (cmp 0)
- return NULL;
+ if (cmp 0) {
+ if (PyErr_Occurred()==PyExc_UnicodeDecodeError)
{
+ PyErr_Clear();
+ }
+ else
+ return NULL;
+}
if (ep0 == mp-ma_table ep-me_key == startkey) {
if (cmp 0)
return ep;
@@ -289,8 +295,13 @@
if (ep-me_hash == hash ep-me_key != dummy) {
startkey = ep-me_key;
cmp = PyObject_RichCompareBool(startkey, key, Py_EQ);
- if (cmp 0)
- return NULL;
+ if (cmp 0) {
+ if (PyErr_Occurred()==PyExc_UnicodeDecodeError)
{
+ PyErr_Clear();
+ }
+ else
+ return NULL;
+}
if (ep0 == mp-ma_table ep-me_key == startkey) {
if (cmp 0)
return ep;
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Dict suppressing exceptions
It wasn't my idea to stop ignoring exceptions in dict lookups; I would gladly have put this off until Py3k, where the main problem (str-unicode __eq__ raising UnicodeError) will go away. But since people are adamant that they want this in sooner, Is this true for dictionaries specifically? Would there really be strong objections to continuing to swallow any Exception (not BaseException) raised by __eq__ ? Writing an __eq__ method that doesn't handle generic objects is sloppy, but it does happen, and I would hate to wrap all dictionaries of objects. -jJ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
