Re: [Python-Dev] Dict suppressing exceptions
M.-A. Lemburg wrote: > M.-A. Lemburg wrote: >> Guido van Rossum wrote: >>> Marc-Andre, how's the patch coming along? >> I'm working on it. >> >> Since we only want equal compares to generate the warning, >> I have to add a rich compare function to Unicode objects. > > Here's an initial version: > > http://sourceforge.net/tracker/index.php?func=detail&aid=1538956&group_id=5470&atid=305470 > > The title of the patch is slightly incorrect - SF doesn't allow > more descriptive titles... :-( FYI: I've checked in the patch as revision 51276. Please give it some serious testing. All tests pass, but there may still be some corner cases left which the tests don't catch. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 14 2006) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Jim Jewett schrieb: > hash was just changed to support the common use of id, even though > some people argued it was *really* a bug in the classes themselves. > This is a similar situation. Similar still different. In the case of __hash__ returning id(), it is very clear what the intention of the program is, and it is implementable to give the program its intended meaning (i.e. use the object address as the source of the hash value). In this case, it is not clear what the intention of the program is. __cmp__ raises an exception, and you say the intention is that this exception should be discarded? Errors should never pass silently, unless explicitly silenced. Some users will find "Oops, I didn't know all these years I had this error there", and correct their code. Some will say "why did they break my program", and correct their code - the corrected code will run just fine on older versions of Python as well. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Guido van Rossum schrieb: >> >__eq__ methods should always return True or False. They should >> >only raise an exception if there is some internal error within >> >one of the objects being compared -- they should never raise >> >an exception because the other object is of an unexpected type. >> >> That policy is currently difficult to implement, but reasonable >> (difficult because it is quite some code to write). > > Why? Are you thinking of the standard library, or of an end user's > __eq__ method? Returning False from your __eq__ if other's type is > unexpected doesn't seem a lot of code. Or am I misunderstanding > something? It seemed like a lot of code to me: In the specific enum example, I first wrote def __eq__(self, other): return self.EnumType is other.EnumType \ and self.__value==other.__value So this was wrong, as it did not take into account 'other' being something completely different, and I wrote def __eq__(self, other): return type(self) is type(other) \ and self.EnumType is other.EnumType \ and self.__value==other.__value Having two additional continuation lines seems quite difficult to me, yet a version that breaks the expression into multiple statements is even longer def __eq__(self, other): if type(self) is not type(other): return False if self.EnumType is not other.EnumType: return False return self.__value==other.__value Compare this to the current two-line __cmp__ implementation: def __cmp__(self, other): assert self.Enumtype is other.EnumType return cmp(self.__value, other.__value) This covers all comparisons just fine in two lines of method body; to implement the __eq__ policy, you need another 6 lines. For consistency, you should also implement __ne__, probably as def __ne__(self, other): return not self.__eq__(other) I expect many people get this wrong, for example http://pyref.infogami.com/__eq__ UserList.py Also, there is confusion as to whether NotImplemented should ever be returned in these. decimal.py believes it does (for different types), sets.py believes it doesn't. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/11/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > Michael Chermside schrieb: > > I propose that we institute a new policy. The policy should state: > > > >__eq__ methods should always return True or False. They should > >only raise an exception if there is some internal error within > >one of the objects being compared -- they should never raise > >an exception because the other object is of an unexpected type. > > That policy is currently difficult to implement, but reasonable > (difficult because it is quite some code to write). Why? Are you thinking of the standard library, or of an end user's __eq__ method? Returning False from your __eq__ if other's type is unexpected doesn't seem a lot of code. Or am I misunderstanding something? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/11/06, Michael Chermside <[EMAIL PROTECTED]> wrote: > Martin v. Löwis writes: > > Now I looked at it, and think that the recipe is broken. > > Not broken, but perhaps wrongheaded. The recipe went out of its way > to ensure that it would raise an exception of enum values from different > enumerations were compared. There's nothing out there saying that this > is a bad thing to do. Except a little voice in my head. :-) > I propose that we institute a new policy. The policy should state: > > __eq__ methods should always return True or False. They should > only raise an exception if there is some internal error within > one of the objects being compared -- they should never raise > an exception because the other object is of an unexpected type. That's pretty much what I want to do in py3k. There, ordering will raise a TypeError by default, but == compares object identity by default, and it's strongly recommended to return False for any unrecognized type. I want the recommendation to be a *little* more relaxed than your proposal; IMO it's fine to raise an exception when your __eq__ implementation is incomplete, i.e. you haven't thought enough about how your class should compare to other classes. The "don't silence exceptions" rule will then cause loud complaints when your class gets compared to something it didn't expect being compared to, signalling that you have more work to do. (And yes, this can cause sleeper bombs when you use these objects as dict keys together with types that it doesn't understand, and the bomb will go off if there's a hash collision; but I've got fairly controlled situations in mind, not robust libraries or frameworks.) > On the other hand, __lt__, __gt__ and friends SHOULD raise an > exception when the object being compared is of some type for > which the ordering does not make sense (e.g.: unicode vs > byte-string or complex vs anything). Yes, that's the py3k rule. > I think we should note this policy someplace official -- perhaps > in the Language Reference where __eq__ and __lt__ are defined. But > I do not think that these changes should be made until Py3K. Fine to update the docs now. > What do others think? Is this the "right" approach? Yes. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Michael Chermside schrieb: > Not broken, but perhaps wrongheaded. The recipe went out of its way > to ensure that it would raise an exception of enum values from different > enumerations were compared. There's nothing out there saying that this > is a bad thing to do. And it's actually fine for three-way compare: different enums don't naturally arrange. > I propose that we institute a new policy. The policy should state: > >__eq__ methods should always return True or False. They should >only raise an exception if there is some internal error within >one of the objects being compared -- they should never raise >an exception because the other object is of an unexpected type. That policy is currently difficult to implement, but reasonable (difficult because it is quite some code to write). >On the other hand, __lt__, __gt__ and friends SHOULD raise an >exception when the object being compared is of some type for >which the ordering does not make sense (e.g.: unicode vs >byte-string or complex vs anything). Right; same for three-way compare. > What do others think? Is this the "right" approach? For the moment, the first section gets augmented by "issue a warning if you think the user is comparing things incorrectly". Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
M.-A. Lemburg wrote: > Guido van Rossum wrote: >> Marc-Andre, how's the patch coming along? > > I'm working on it. > > Since we only want equal compares to generate the warning, > I have to add a rich compare function to Unicode objects. Here's an initial version: http://sourceforge.net/tracker/index.php?func=detail&aid=1538956&group_id=5470&atid=305470 The title of the patch is slightly incorrect - SF doesn't allow more descriptive titles... :-( -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 11 2006) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Martin v. Löwis writes: > Now I looked at it, and think that the recipe is broken. Not broken, but perhaps wrongheaded. The recipe went out of its way to ensure that it would raise an exception of enum values from different enumerations were compared. There's nothing out there saying that this is a bad thing to do. I propose that we institute a new policy. The policy should state: __eq__ methods should always return True or False. They should only raise an exception if there is some internal error within one of the objects being compared -- they should never raise an exception because the other object is of an unexpected type. On the other hand, __lt__, __gt__ and friends SHOULD raise an exception when the object being compared is of some type for which the ordering does not make sense (e.g.: unicode vs byte-string or complex vs anything). I think we should note this policy someplace official -- perhaps in the Language Reference where __eq__ and __lt__ are defined. But I do not think that these changes should be made until Py3K. What do others think? Is this the "right" approach? -- Michael Chermside ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Dict suppressing exceptions
Martin v. Löwis wrote: > Now I looked at it, and think that the recipe is broken. It should > add an __eq__ method which is > def __eq__(self, other): > return type(self) is type(other) \ > and self.EnumType is other.EnumType \ > and self.__value==other.__value) Absolutely. But the point is that there are a lot of these broken objects out there; this one was on a 5-star recipe that has been around for a while, and still no one caught it. That suggests the bug could reasonably be in any 3rd-party library. The existence of a (common) bug in someone else's code shouldn't keep me from using a dictionary of objects. hash was just changed to support the common use of id, even though some people argued it was *really* a bug in the classes themselves. This is a similar situation. -jJ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Michael Chermside schrieb: > I don't *strongly* object to this consensus, but if you haven't > glanced at my original example, take a look - it might convince you. > The proposed solution will not help with my example. I ignored your example the first time because it was too complicated to understand. Now I looked at it, and think that the recipe is broken. It should add an __eq__ method which is def __eq__(self, other): return type(self) is type(other) \ and self.EnumType is other.EnumType \ and self.__value==other.__value) Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Guido van Rossum schrieb: > Me too, and that's what we'll do in py3k. But in 2.5, we're bound by > the decisions we made in 1999-2000 about unicode. (Unless Martin has a > convincing reason not to have a warning?) Only the general anti-warning argument: it's not the developer which will see the warning, but the end user. In this case, they might easily get hundreds of warning in a short time, and these fill their Apache log files. They complain about Python eating their disk space. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Guido van Rossum wrote: > Marc-Andre, how's the patch coming along? I'm working on it. Since we only want equal compares to generate the warning, I have to add a rich compare function to Unicode objects. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 11 2006) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/11/06, Armin Rigo <[EMAIL PROTECTED]> wrote: > Hi, > > On Thu, Aug 10, 2006 at 02:36:16PM -0700, Guido van Rossum wrote: > > > On Thu, Aug 10, 2006 at 09:11:42PM +0200, "Martin v. L?wis" wrote: > > > > I'm in favour of having this __eq__ just return False. I don't think > > > > the warning is necessary, (...) > > > > > > +1 > > > > Can you explain why you believe that no warning is necessary? > > Ah... mostly out of ignorance, I fear. I did not realize there were > strange cases like u"\xff" == "\xff". I will leave this to more > unicode-minded people to decide, but I'm still definitely of the idea > that the __eq__ should not raise an exception. Me too, and that's what we'll do in py3k. But in 2.5, we're bound by the decisions we made in 1999-2000 about unicode. (Unless Martin has a convincing reason not to have a warning?) Marc-Andre, how's the patch coming along? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Guido writes: > Alas, I have no idea what it does. Can you come up with an example > that doesn't require enums and localization? Sorry. Here's the short version: Fact 1: Sometimes people create objects that raise exceptions when compared for equality. Maybe it's a bad idea to do this, and objects should never raise exceptions when compared except to signal internal inconsistancy -- but the practice is common enough to show up in a 5-star recipe for enums in the Cookbook. Fact 2: Unrelated objects are often put in dictionarys together. I used a "localization dictionary" as an example. I feel this is a legitimate practice. If dictionarys stop suppressing exceptions raised by equality tests then those two facts aren't compatible. Programs that used to work fine will break with 2.4. I'm OK with your proposed solution (fix the case of str-unicode but let everyone else using objects that raise exceptions suffer), but I think it would be more friendly to use warnings for now and exceptions after another full release cycle. The warnings should solve the underlying issue (hard-to-debug problems). -- Michael Chermside ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Armin Rigo wrote: > Hi, > > On Thu, Aug 10, 2006 at 02:36:16PM -0700, Guido van Rossum wrote: >>> On Thu, Aug 10, 2006 at 09:11:42PM +0200, "Martin v. L?wis" wrote: I'm in favour of having this __eq__ just return False. I don't think the warning is necessary, (...) >>> +1 >> Can you explain why you believe that no warning is necessary? > > Ah... mostly out of ignorance, I fear. I did not realize there were > strange cases like u"\xff" == "\xff". This is not all that strange. Perhaps this example looks closer to life ;-) if u'D\xfcsseldorf' != 'D\xfcsseldorf': move_to_Cologne() Now, we wouldn't want that to go through without a warning, do we... ;-) [Background: there's a bit of rivalry between those two cities; see e.g. see http://en.wikipedia.org/wiki/D%C3%BCsseldorf] -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 11 2006) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Hi, On Thu, Aug 10, 2006 at 02:36:16PM -0700, Guido van Rossum wrote: > > On Thu, Aug 10, 2006 at 09:11:42PM +0200, "Martin v. L?wis" wrote: > > > I'm in favour of having this __eq__ just return False. I don't think > > > the warning is necessary, (...) > > > > +1 > > Can you explain why you believe that no warning is necessary? Ah... mostly out of ignorance, I fear. I did not realize there were strange cases like u"\xff" == "\xff". I will leave this to more unicode-minded people to decide, but I'm still definitely of the idea that the __eq__ should not raise an exception. A bientot, Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
M.-A. Lemburg wrote: > Michael Chermside wrote: >>> How about we change unicode-vs-str __eq__ to >>> issue a warning (and return False) instead of raising >>> UnicodeException? >> [... Marc-Andre Lemburg agrees ...] >>> Great! Now we need someone to volunteer to write a patch (which should >>> include doc and NEWS updates) in time for beta 3 (Aug 18). >> I don't *strongly* object to this consensus, but if you haven't >> glanced at my original example, take a look - it might convince you. >> The proposed solution will not help with my example. I'm not sure the >> motivation for breaking code like that example -- the bug-hunting >> motivation is satisfied by issuing a warning as Michael Urman proposes, >> then use an exception after one more release when people have had time >> to fix their code. > > The warning framework gives programmers great flexibility in handling > these situation: > > * they can silence the warning, just report it once, always report it > * they can have the warning raise an exception > * they can log the warning in some way > > It's very flexible to accommodate for all kinds of ways to handle > the situation. > > By introducing a new category of warning for these Unicode-related > warnings, adding specific warning filters will be easy. This sounds like the correct tool to me. +1 Would it be possible to generate warnings when either Unicode or stings are coerced to the other implicitly but not explicitly? That may also generate a warning in the case of the dictionary problem being discussed, (I think), and may be more useful for checking for non-intentionally mixed Unicode and strings. Ron ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/10/06, Armin Rigo <[EMAIL PROTECTED]> wrote: > Hi, > > On Thu, Aug 10, 2006 at 09:11:42PM +0200, "Martin v. L?wis" wrote: > > I'm in favour of having this __eq__ just return False. I don't think > > the warning is necessary, (...) > > +1 Can you explain why you believe that no warning is necessary? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/10/06, "Martin v. Löwis" <[EMAIL PROTECTED]> wrote: > Guido van Rossum schrieb: > > Hmm... Here's an idea... How about we change unicode-vs-str __eq__ to > > issue a warning (and return False) instead of raising > > UnicodeException? > > I'm in favour of having this __eq__ just return False. I don't think > the warning is necessary, and believe that people will complain about > getting it, but if that is the only way to achieve consensus, > so be it. Not sure I agree. I think that users would find it disturbing (and it could hide bugs) if u"\xff" == "\xff" returned False without warning while u"\x7f" == "\x7f" returns True. > > A warning would arguably have the same helping effect. (I suspect > > actually that we would have used warnings all along except we didn't > > have the warning framework when unicode was first introduced.) > > I believe we would have neither had we rich comparisons available > at the time. That unicode-vs-str-__eq__ raises an exception is just > a fall-out of not supporting __cmp__ (and it is good that __cmp__ > is not supported - there is no meaningful way to order str/unicode > if the string is not in the system encoding). Maybe. I agree that 3-way compare has no business guessing the ordering of u"\xff" and "\xff". I'm not so sure that if we'd had rich comparison we'd have chosen __eq__ to return False for these. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Guido van Rossum wrote: > On 8/10/06, M.-A. Lemburg <[EMAIL PROTECTED]> wrote: >> Guido van Rossum wrote: >>> Hmm... Here's an idea... How about we change unicode-vs-str __eq__ to >>> issue a warning (and return False) instead of raising >>> UnicodeException? That won't break much code (it's unlikely that >>> people *depend* on this exception since it's generally a symptom of >>> insane mixing of str and unicode). Then no further changes to >>> dictobject.c are necessary (except fixing that one misleading >>> comment). >> Good idea. > > Great! Now we need someone to volunteer to write a patch (which should > include doc and NEWS updates) in time for beta 3 (Aug 18). I'll give it a go. Not sure what kinds of docs we'll need for this, though (and more importantly: where to put them, e.g. into the tutorial in the dictionary section, somewhere into the Unicode section ?) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 10 2006) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Michael Chermside wrote: >> How about we change unicode-vs-str __eq__ to >> issue a warning (and return False) instead of raising >> UnicodeException? > [... Marc-Andre Lemburg agrees ...] >> Great! Now we need someone to volunteer to write a patch (which should >> include doc and NEWS updates) in time for beta 3 (Aug 18). > > I don't *strongly* object to this consensus, but if you haven't > glanced at my original example, take a look - it might convince you. > The proposed solution will not help with my example. I'm not sure the > motivation for breaking code like that example -- the bug-hunting > motivation is satisfied by issuing a warning as Michael Urman proposes, > then use an exception after one more release when people have had time > to fix their code. The warning framework gives programmers great flexibility in handling these situation: * they can silence the warning, just report it once, always report it * they can have the warning raise an exception * they can log the warning in some way It's very flexible to accommodate for all kinds of ways to handle the situation. By introducing a new category of warning for these Unicode-related warnings, adding specific warning filters will be easy. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 10 2006) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/10/06, Michael Chermside <[EMAIL PROTECTED]> wrote: > See also my example from the beginning of this thread > (http://mail.python.org/pipermail/python-dev/2006-August/067978.html). > The example wasn't from real code, but it WAS quite plausible -- > straightforward use of a popular Python Cookbook recipe. [...] > I don't *strongly* object to this consensus, but if you haven't > glanced at my original example, take a look - it might convince you. Alas, I have no idea what it does. Can you come up with an example that doesn't require enums and localization? -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Hi, On Thu, Aug 10, 2006 at 09:11:42PM +0200, "Martin v. L?wis" wrote: > I'm in favour of having this __eq__ just return False. I don't think > the warning is necessary, (...) +1 Armin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Michael Urman writes: > I strongly believe that unicode vs str here is the symptom and not the > actual problem. The comparison between two non-us-ascii str/unicode > instances is but one of many ways to raise an exception during > comparison. [... example ...] > Yes this is made up code. But I'm not arguing for a feature; I'm > arguing for backwards compatibility. Because we do not know where > these legitimate uses are, we cannot evaluate their likelihood to > exist nor the level of breakage they will cause. If we make this a > warning for any exception, we can satisfy both imagined camps. Those > in Armin's position can make that warning raise an exception while > debugging, and those using it on purpose can squash it. See also my example from the beginning of this thread (http://mail.python.org/pipermail/python-dev/2006-August/067978.html). The example wasn't from real code, but it WAS quite plausible -- straightforward use of a popular Python Cookbook recipe. But we seem to have reached a rough consensus: Later, Guido writes: > How about we change unicode-vs-str __eq__ to > issue a warning (and return False) instead of raising > UnicodeException? [... Marc-Andre Lemburg agrees ...] > Great! Now we need someone to volunteer to write a patch (which should > include doc and NEWS updates) in time for beta 3 (Aug 18). I don't *strongly* object to this consensus, but if you haven't glanced at my original example, take a look - it might convince you. The proposed solution will not help with my example. I'm not sure the motivation for breaking code like that example -- the bug-hunting motivation is satisfied by issuing a warning as Michael Urman proposes, then use an exception after one more release when people have had time to fix their code. -- Michael Chermside ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Guido van Rossum schrieb: > Hmm... Here's an idea... How about we change unicode-vs-str __eq__ to > issue a warning (and return False) instead of raising > UnicodeException? I'm in favour of having this __eq__ just return False. I don't think the warning is necessary, and believe that people will complain about getting it, but if that is the only way to achieve consensus, so be it. > A warning would arguably have the same helping effect. (I suspect > actually that we would have used warnings all along except we didn't > have the warning framework when unicode was first introduced.) I believe we would have neither had we rich comparisons available at the time. That unicode-vs-str-__eq__ raises an exception is just a fall-out of not supporting __cmp__ (and it is good that __cmp__ is not supported - there is no meaningful way to order str/unicode if the string is not in the system encoding). Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/10/06, M.-A. Lemburg <[EMAIL PROTECTED]> wrote: > Guido van Rossum wrote: > > Hmm... Here's an idea... How about we change unicode-vs-str __eq__ to > > issue a warning (and return False) instead of raising > > UnicodeException? That won't break much code (it's unlikely that > > people *depend* on this exception since it's generally a symptom of > > insane mixing of str and unicode). Then no further changes to > > dictobject.c are necessary (except fixing that one misleading > > comment). > > Good idea. Great! Now we need someone to volunteer to write a patch (which should include doc and NEWS updates) in time for beta 3 (Aug 18). -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Guido van Rossum wrote: > On 8/10/06, M.-A. Lemburg <[EMAIL PROTECTED]> wrote: >> I'd suggest that we still inform the programmers of the problem >> by issuing a warning (which they can then silence at will), >> maybe a new PyExc_UnicodeWarning. > > Hmm... Here's an idea... How about we change unicode-vs-str __eq__ to > issue a warning (and return False) instead of raising > UnicodeException? That won't break much code (it's unlikely that > people *depend* on this exception since it's generally a symptom of > insane mixing of str and unicode). Then no further changes to > dictobject.c are necessary (except fixing that one misleading > comment). Good idea. >> Note that these exceptions help programmers making their applications >> Unicode compatible, so silencing them completely would remove the >> possibility to detect the case of mixing strings and Unicode as >> keys in a dictionary. > > A warning would arguably have the same helping effect. (I suspect > actually that we would have used warnings all along except we didn't > have the warning framework when unicode was first introduced.) Probably, yes. These days, I believe that we should have not gone for the string-Unicode integration magic at all and instead done the same of what's planned for Py3k. Sometimes a clean cut is better than trying to muddle along. >> BTW, in Py3k, this case would not trigger at all, since all text >> would be Unicode and bytes wouldn't be comparable to Unicode >> anyway. However, that's a different discussion which we can have >> after Python 2.5 is out the door. > > Which is why I said I would have gladly waited until Py3k to fix this. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 10 2006) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/10/06, M.-A. Lemburg <[EMAIL PROTECTED]> wrote: > I'd suggest that we still inform the programmers of the problem > by issuing a warning (which they can then silence at will), > maybe a new PyExc_UnicodeWarning. Hmm... Here's an idea... How about we change unicode-vs-str __eq__ to issue a warning (and return False) instead of raising UnicodeException? That won't break much code (it's unlikely that people *depend* on this exception since it's generally a symptom of insane mixing of str and unicode). Then no further changes to dictobject.c are necessary (except fixing that one misleading comment). > Note that these exceptions help programmers making their applications > Unicode compatible, so silencing them completely would remove the > possibility to detect the case of mixing strings and Unicode as > keys in a dictionary. A warning would arguably have the same helping effect. (I suspect actually that we would have used warnings all along except we didn't have the warning framework when unicode was first introduced.) > BTW, in Py3k, this case would not trigger at all, since all text > would be Unicode and bytes wouldn't be comparable to Unicode > anyway. However, that's a different discussion which we can have > after Python 2.5 is out the door. Which is why I said I would have gladly waited until Py3k to fix this. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/10/06, Michael Urman <[EMAIL PROTECTED]> wrote: > I strongly believe that unicode vs str here is the symptom and not the > actual problem. No. Everywhere when __eq__ fails, we can safely tell the user that it's a bug in their __eq__ that they should fix (maybe by making it return False when the other object isn't a type they recognize). But we can't fix unicode-vs-str comparison without breaking too much code. So we have to somehow deal with that without changing the behavior of that particular __eq__. -- --Guido van Rossum (home page: http://www.python.org/~guido/) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
On 8/10/06, M.-A. Lemburg <[EMAIL PROTECTED]> wrote:
> Guido van Rossum wrote:
> > But since people are adamant that they want this in sooner, I suggest
> > that to minimize breakage we could make an exception for str-unicode
> > comparisons.
> > What do people think?
>
> I'd suggest that we still inform the programmers of the problem
> by issuing a warning (which they can then silence at will),
> maybe a new PyExc_UnicodeWarning.
>
> BTW, in Py3k, this case would not trigger at all, since all text
> would be Unicode and bytes wouldn't be comparable to Unicode
> anyway. However, that's a different discussion which we can have
> after Python 2.5 is out the door.
I strongly believe that unicode vs str here is the symptom and not the
actual problem. The comparison between two non-us-ascii str/unicode
instances is but one of many ways to raise an exception during
comparison. It's not even obvious ahead of time when it will occur.
Try my example below with (sys.maxint << 1) and -2 instead of 1 and 1.
Do you expect a problem?
Because str/unicode is the common case, we saw it first. If we address
the symptom instead of the problem, someone will be complaining within
a years time because they have a class that they mix in with other
items for a function handler lookup, or who knows what, that works
like the following:
>>> class hasher(object):
... def __init__(self, hashval):
... self.hashval = hashval
... def __hash__(self):
... return hash(self.hashval)
... def __eq__(self, o):
... if not isinstance(o, hasher):
... raise TypeError("Cannot compare hashval to non hashval")
... return self.hashval == o.hashval
in python2.4:
>>> dict.fromkeys([1, hasher(1)])
{1: None, <__main__.hasher object at 0xa7a5326c>: None}
in python2.5b2:
>>> dict.fromkeys([1, hasher(1)])
Traceback (most recent call last):
File "", line 1, in
File "", line 8, in __eq__
TypeError: Cannot compare hashval to non hashval
Yes this is made up code. But I'm not arguing for a feature; I'm
arguing for backwards compatibility. Because we do not know where
these legitimate uses are, we cannot evaluate their likelihood to
exist nor the level of breakage they will cause. If we make this a
warning for any exception, we can satisfy both imagined camps. Those
in Armin's position can make that warning raise an exception while
debugging, and those using it on purpose can squash it.
I understand the utility of being able to see this case happening. I'm
not sure it's worth the warning. I'm positive it's not worth the
backwards incompatibility of the raised exception.
Michael
--
Michael Urman http://www.tortall.net/mu/blog
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Guido van Rossum wrote: > > - the exception could be narrowed even further by only suppressing the > exception when startkey and key are both either str or unicode > instances. > I always assumed dictionaries would work exactly like this. So, at least it would now work as I had always expected (and others possibly too). - Ralf ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Dict suppressing exceptions
Guido van Rossum wrote:
> I've been happily ignoring python-dev for the last three weeks or so,
> and Neal just pointed me to some thorny issues that are close to
> resolution but not quite yet resolved, yet need to be before beta 3 on
> August 18 (Friday next week).
>
> Here's my take on the dict-suppressing-exceptions issue (I'll send out
> separate messages for each issue where Neal has asked me to weigh in).
>
> It wasn't my idea to stop ignoring exceptions in dict lookups; I would
> gladly have put this off until Py3k, where the main problem
> (str-unicode __eq__ raising UnicodeError) will go away.
>
> But since people are adamant that they want this in sooner, I suggest
> that to minimize breakage we could make an exception for str-unicode
> comparisons.
>
> I came up with the following code to reproduce the issue; this prints
> 0 in 2.2, False in 2.3 and 2.4, but raises UnicodeDecodeError in 2.5
> (head):
>
> a = {u"1": 1}
> x = hash(u"1")
> class C(str):
> def __hash__(s): return x
> print C("\xff") in a
>
> The following patch makes this print False in 2.5 again.
>
> Notes about the patch:
>
> - this also fixes an out-of-date comment that should be fixed even if
> the rest of the idea is rejected (lookdict_string() can return NULL
> when it calls lookdict)
>
> - the exception could be narrowed even further by only suppressing the
> exception when startkey and key are both either str or unicode
> instances.
>
> What do people think?
I'd suggest that we still inform the programmers of the problem
by issuing a warning (which they can then silence at will),
maybe a new PyExc_UnicodeWarning.
Note that these exceptions help programmers making their applications
Unicode compatible, so silencing them completely would remove the
possibility to detect the case of mixing strings and Unicode as
keys in a dictionary.
BTW, in Py3k, this case would not trigger at all, since all text
would be Unicode and bytes wouldn't be comparable to Unicode
anyway. However, that's a different discussion which we can have
after Python 2.5 is out the door.
> --- Objects/dictobject.c (revision 51180)
> +++ Objects/dictobject.c (working copy)
> @@ -230,7 +230,8 @@
> lookdict() is general-purpose, and may return NULL if (and only if) a
> comparison raises an exception (this was new in Python 2.5).
> lookdict_string() below is specialized to string keys, comparison of which
> can
> -never raise an exception; that function can never return NULL. For both,
> when
> +never raise an exception; that function can never return NULL (except when it
> +decides to replace itself with the more general lookdict()). For both, when
> the key isn't found a dictentry* is returned for which the me_value field is
> NULL; this is the slot in the dict at which the key would have been found,
> and
> the caller can (if it wishes) add the pair to the returned
> @@ -259,8 +260,13 @@
> if (ep->me_hash == hash) {
> startkey = ep->me_key;
> cmp = PyObject_RichCompareBool(startkey, key, Py_EQ);
> - if (cmp < 0)
> - return NULL;
> + if (cmp < 0) {
> + if (PyErr_Occurred()==PyExc_UnicodeDecodeError)
> {
if (PyErr_Warn(PyExc_UnicodeWarning,
"mixing unicode and strings "
"as dictionary keys") < 0)
return NULL;
> +PyErr_Clear();
> + }
> + else
> + return NULL;
> +}
> if (ep0 == mp->ma_table && ep->me_key == startkey) {
> if (cmp > 0)
> return ep;
> @@ -289,8 +295,13 @@
> if (ep->me_hash == hash && ep->me_key != dummy) {
> startkey = ep->me_key;
> cmp = PyObject_RichCompareBool(startkey, key, Py_EQ);
> - if (cmp < 0)
> - return NULL;
> + if (cmp < 0) {
> + if (PyErr_Occurred()==PyExc_UnicodeDecodeError)
> {
> + PyErr_Clear();
> + }
> + else
> + return NULL;
> +}
> if (ep0 == mp->ma_table && ep->me_key == startkey) {
> if (cmp > 0)
> return ep;
>
>
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Aug 10 2006)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...
Re: [Python-Dev] Dict suppressing exceptions
"Jim Jewett" <[EMAIL PROTECTED]> writes: >> It wasn't my idea to stop ignoring exceptions in dict lookups; I would >> gladly have put this off until Py3k, where the main problem >> (str-unicode __eq__ raising UnicodeError) will go away. > >> But since people are adamant that they want this in sooner, > > Is this true for dictionaries specifically? > > Would there really be strong objections to continuing to swallow > any Exception (not BaseException) raised by __eq__ ? Armin's reason for changing dictionaries in this way was that enormous debugging pain was caused by dicts swallowing exceptions raised by __eq__ methods. Having the __eq__ methods swallow the exceptions by themselves would make the situation *worse*, not better. Cheers, mwh -- * vegai wears his reading bra. umm, I mean glasses -- from Twisted.Quotes ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Dict suppressing exceptions
> It wasn't my idea to stop ignoring exceptions in dict lookups; I would > gladly have put this off until Py3k, where the main problem > (str-unicode __eq__ raising UnicodeError) will go away. > But since people are adamant that they want this in sooner, Is this true for dictionaries specifically? Would there really be strong objections to continuing to swallow any Exception (not BaseException) raised by __eq__ ? Writing an __eq__ method that doesn't handle generic objects is sloppy, but it does happen, and I would hate to wrap all dictionaries of objects. -jJ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Dict suppressing exceptions
I've been happily ignoring python-dev for the last three weeks or so,
and Neal just pointed me to some thorny issues that are close to
resolution but not quite yet resolved, yet need to be before beta 3 on
August 18 (Friday next week).
Here's my take on the dict-suppressing-exceptions issue (I'll send out
separate messages for each issue where Neal has asked me to weigh in).
It wasn't my idea to stop ignoring exceptions in dict lookups; I would
gladly have put this off until Py3k, where the main problem
(str-unicode __eq__ raising UnicodeError) will go away.
But since people are adamant that they want this in sooner, I suggest
that to minimize breakage we could make an exception for str-unicode
comparisons.
I came up with the following code to reproduce the issue; this prints
0 in 2.2, False in 2.3 and 2.4, but raises UnicodeDecodeError in 2.5
(head):
a = {u"1": 1}
x = hash(u"1")
class C(str):
def __hash__(s): return x
print C("\xff") in a
The following patch makes this print False in 2.5 again.
Notes about the patch:
- this also fixes an out-of-date comment that should be fixed even if
the rest of the idea is rejected (lookdict_string() can return NULL
when it calls lookdict)
- the exception could be narrowed even further by only suppressing the
exception when startkey and key are both either str or unicode
instances.
What do people think?
--- Objects/dictobject.c(revision 51180)
+++ Objects/dictobject.c(working copy)
@@ -230,7 +230,8 @@
lookdict() is general-purpose, and may return NULL if (and only if) a
comparison raises an exception (this was new in Python 2.5).
lookdict_string() below is specialized to string keys, comparison of which can
-never raise an exception; that function can never return NULL. For both, when
+never raise an exception; that function can never return NULL (except when it
+decides to replace itself with the more general lookdict()). For both, when
the key isn't found a dictentry* is returned for which the me_value field is
NULL; this is the slot in the dict at which the key would have been found, and
the caller can (if it wishes) add the pair to the returned
@@ -259,8 +260,13 @@
if (ep->me_hash == hash) {
startkey = ep->me_key;
cmp = PyObject_RichCompareBool(startkey, key, Py_EQ);
- if (cmp < 0)
- return NULL;
+ if (cmp < 0) {
+ if (PyErr_Occurred()==PyExc_UnicodeDecodeError)
{
+ PyErr_Clear();
+ }
+ else
+ return NULL;
+}
if (ep0 == mp->ma_table && ep->me_key == startkey) {
if (cmp > 0)
return ep;
@@ -289,8 +295,13 @@
if (ep->me_hash == hash && ep->me_key != dummy) {
startkey = ep->me_key;
cmp = PyObject_RichCompareBool(startkey, key, Py_EQ);
- if (cmp < 0)
- return NULL;
+ if (cmp < 0) {
+ if (PyErr_Occurred()==PyExc_UnicodeDecodeError)
{
+ PyErr_Clear();
+ }
+ else
+ return NULL;
+}
if (ep0 == mp->ma_table && ep->me_key == startkey) {
if (cmp > 0)
return ep;
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
