Re: Guido rethinking removal of cmp from sort method
In article 4d8be3bb.4030...@v.loewis.de, Martin v. Loewis mar...@v.loewis.de wrote: Martin deleted the attribution for Carl Banks: The cmp argument doesn't depend in any way on an object's __cmp__ method, so getting rid of __cmp__ wasn't any good readon to also get rid of the cmp argument So what do you think about the cmp() builtin? Should have stayed, or was it ok to remove it? If it should have stayed: how should it's implementation have looked like? If it was ok to remove it: how are people supposed to fill out the cmp= argument in cases where they use the cmp() builtin in 2.x? Actually, my take is that removing __cmp__ was a mistake. (I already argued about it back in python-dev before it happened, and I see little point rehashing it. My reason is strictly efficiency grounds: when comparisons are expensive -- such as Decimal object -- __cmp__ is faster.) -- Aahz (a...@pythoncraft.com) * http://www.pythoncraft.com/ Beware of companies that claim to be like a family. They might not be lying. --Jill Lundquist -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 04/09/11 01:08, Aahz wrote: Actually, my take is that removing __cmp__ was a mistake. (I already argued about it back in python-dev before it happened, and I see little point rehashing it. My reason is strictly efficiency grounds: when comparisons are expensive -- such as Decimal object -- __cmp__ is faster.) I don't get you... why would sorting a list using __cmp__ be faster when comparisons are expensive? -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
In article 4d9f32a2$1...@dnews.tpgi.com.au, Lie Ryan lie.1...@gmail.com wrote: On 04/09/11 01:08, Aahz wrote: Actually, my take is that removing __cmp__ was a mistake. (I already argued about it back in python-dev before it happened, and I see little point rehashing it. My reason is strictly efficiency grounds: when comparisons are expensive -- such as Decimal object -- __cmp__ is faster.) I don't get you... why would sorting a list using __cmp__ be faster when comparisons are expensive? Not sorting (because sorting only requires one comparison), but any operation involving multiple comparisons. Consider this: if a b: x() elif a == b: y() else: z() For a = b, you need to make two comparisons. Now consider this: r = cmp(a, b) if r 0: x() elif r == 0: y() else: z() -- Aahz (a...@pythoncraft.com) * http://www.pythoncraft.com/ At Resolver we've found it useful to short-circuit any doubt and just refer to comments in code as 'lies'. :-) --Michael Foord paraphrases Christian Muirhead on python-dev, 2009-03-22 -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Fri, Apr 01, 2011 at 10:21:33PM -0400, Terry Reedy wrote: rewriting cmp_to_key in C is underway http://bugs.python.org/issue11707 Nice to know! Any chance this wil get into 2.7.x? -- Antoon Pardon -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 04/04/11 19:34, Antoon Pardon wrote: On Fri, Apr 01, 2011 at 10:21:33PM -0400, Terry Reedy wrote: rewriting cmp_to_key in C is underway http://bugs.python.org/issue11707 Nice to know! Any chance this wil get into 2.7.x? Python 2.7 still have list.sort(cmp=...)/sorted(cmp=...), so cmp_to_key is not much use there. Just pass your comparison function to the cmp argument. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 4/4/2011 5:34 AM, Antoon Pardon wrote: On Fri, Apr 01, 2011 at 10:21:33PM -0400, Terry Reedy wrote: rewriting cmp_to_key in C is underway http://bugs.python.org/issue11707 Nice to know! Any chance this wil get into 2.7.x? I posted the question to the issue. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 4/4/2011 9:35 AM, Lie Ryan wrote: On 04/04/11 19:34, Antoon Pardon wrote: On Fri, Apr 01, 2011 at 10:21:33PM -0400, Terry Reedy wrote: rewriting cmp_to_key in C is underway http://bugs.python.org/issue11707 Nice to know! Any chance this wil get into 2.7.x? Python 2.7 still have list.sort(cmp=...)/sorted(cmp=...), so cmp_to_key is not much use there. Just pass your comparison function to the cmp argument. .cmp_to_key was added to ease moving to 3.x or maintaining code usable on both. A faster version would encourage such use. But it is not a bug fix exactly, and there is always worry that permanance enhancements may have unforseen side effects. I will let Raymond make the call on this. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
O fix exactly, and there is always worry that permanance enhancements may have unforseen side effects. I will let Raymond make the call on this. /permanance/performance/, /unforseen/unforeseen/ -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Apr 2, 5:13 am, Steven D'Aprano steve +comp.lang.pyt...@pearwood.info wrote: Should I use the list, list2, sortable_list, sortable_list2, sortable_lost3, [note spelling, which we're stuck with forever], heterogeneous_list, heterogeneous_list_without_stooge_sort, new_list, fancy_list, fancy_list2, fancy_list_with_extra_oomph, newer_than_new_list or list3? Each and every interface carries a cost. Even if it is small, that cost must be weighed up against the benefits, rather than declaring that all interfaces are sacred once published. Yes and whilst that was a brilliant display of bombastic arrogance your statements miss the point completely. And to respond i'll pull a sentence from your own post... None of those assumptions are even *remotely* true in the real world. Moving on... Removing a published interface imposes a one-time cost on those using that interface, but it has an on-going benefit for all. Oh really? And what about the large steaming pile of elephant dung in the room your nose seems to be unable to smell? You just *assume* that more new hands are hopping on board the old Python vessel to hell than old hands are dangling on to the rudders for dear life. What a naive assumption Mr. D'Aprano! As we all know Python has experienced an explosion of usage over the past years, however i would say with the confusions of the Python2 to Python3 conversion, old tutorials, Ruby's competition, and just plain mis information in the wild we have cast deep into the throes of a internal Pythonic rejection-ism and now find ourselves teetering on the brink of full blown world wide Pythonic recession-ism unless we get this runaway ship under control very quickly. Now whilst i agree with most of the changes in Python3 i wonder if some of them could have waited until Python4. We seemed to have become so pedantic as to render ourselves blind to the cries of others. And when i say others i am not speaking general others. No, i am speaking of fellow tried and true Pythonistas who have many years of investment in this language. Folks who have vast libraries of Python code that have been rendered useless because a few elites have spent too much time lamenting minutia. Where is the call from on high to rid the web of old Python2.x tuts and ring in the era of Python3.x compatibility? Where is the energy to re-ignite the flames of collaborative community? Where is the leadership this community desperately needs? Where is the excitement for the future of Python in the old hats? Oh yes i forgot... they're too busy porting 2.x code to give a flying fig! -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Tue, Apr 5, 2011 at 7:10 AM, rantingrick rantingr...@gmail.com wrote: olks who have vast libraries of Python code that have been rendered useless because a few elites have spent too much time lamenting minutia. How is the code rendered useless when (a) Python 2.7 is still the default Python in many places, and (b) in any place where it's not, it's going to be about one command to install a python2? ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
rantingrick wrote: Yes and whilst that was a brilliant display of bombastic arrogance your statements miss the point completely. And what about the large steaming pile of elephant dung in the room your nose seems to be unable to smell? As we all know Python has experienced an explosion of usage over the past years, however i would say with the confusions of the Python2 to Python3 conversion, old tutorials, Ruby's competition, and just plain mis information in the wild we have cast deep into the throes of a internal Pythonic rejection-ism and now find ourselves teetering on the brink of full blown world wide Pythonic recession-ism unless we get this runaway ship under control very quickly. oooh, ouch. Sadly, there may be some truth in there... ... to play the advocate for a moment, as the client community we need to get our heads around the motives of the development community. The overall goal (it seems) is to make the Python language an 'ideal' that is admittantly evolving over time. The frustrating thing for the rest of us is that we now must develop code for two versions--- as well we must not make assumptions about whether our code will port nicely to other systems/platforms. While this is frustrating, it is not an insurmountable mountain of elephant dung... although, I found the metaphor funny. :) Python(3) is a new language. It has many of the same characteristics of Python2, but will be more consistent, cleaner, leaner, more robust... and certainly loved more universally by more people the world over for centuries to come;-) Bring out yer dead...,Bring out yer dead... :) -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Apr 4, 4:41 pm, Chris Angelico ros...@gmail.com wrote: How is the code rendered useless when (a) Python 2.7 is still the default Python in many places, That's the point. We are going to see Python2.x around for a very long time. A *very* long time Chris. Sadly if this conversion was planned a wee bit better, we could have seen it happen rather quickly instead. We could have reined in the multiplicity instead of propagating it! and (b) in any place where it's not, it's going to be about one command to install a python2? Oh thanks Chris for revealing the simplicity of 2 to 3 code porting. And might ask where you will begin to volunteer your expertise to the gazillion lines of code that need porting? hmm? -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Tue, Apr 5, 2011 at 8:16 AM, rantingrick rantingr...@gmail.com wrote: and (b) in any place where it's not, it's going to be about one command to install a python2? Oh thanks Chris for revealing the simplicity of 2 to 3 code porting. And might ask where you will begin to volunteer your expertise to the gazillion lines of code that need porting? hmm? That is specifically NOT porting. Sorry, I'm not a 2to3-on-call. I just code for one platform. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Mon, 04 Apr 2011 17:09:07 -0500, harrismh777 wrote: Python(3) is a new language. It has many of the same characteristics of Python2, but will be more consistent, cleaner, leaner, more robust... and certainly loved more universally by more people the world over for centuries to come;-) Only if you define language so narrowly that Python 2.1 and Python 2.2 are also different languages, or CPython and Jython or IronPython. I prefer to consider Python 2.7 and Python 3.x as different dialects of the same language. There are a very few handful of incompatibilities, most of which can be automatically resolved by the 2to3 fixers. To describe them as different languages leaves no term to describe the differences between (say) Python and Cobra: http://cobra-language.com/docs/python/ let alone something like Python vs Forth. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Tue, 05 Apr 2011 07:41:37 +1000, Chris Angelico wrote: On Tue, Apr 5, 2011 at 7:10 AM, rantingrick rantingr...@gmail.com wrote: olks who have vast libraries of Python code that have been rendered useless because a few elites have spent too much time lamenting minutia. How is the code rendered useless when (a) Python 2.7 is still the default Python in many places, and (b) in any place where it's not, it's going to be about one command to install a python2? Please don't feed the troll. Check the archives. rantingrick isn't interested in good-faith debate. He's trying to rally followers to lead on his crusade to save Python from itself (which *entirely* consists of him declaring that there is a problem, and everyone else dropping what they're doing to do the actual work of fixing it). -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Steven D'Aprano wrote: I prefer to consider Python 2.7 and Python 3.x as different dialects of the same language. There are a very few handful of incompatibilities, most of which can be automatically resolved by the 2to3 fixers. Yes, I am actually finding this to be consistent with my experience of trying to come up to speed with 3.2. I have been relieved to find that less has changed than the fear-mongering and bickering was leading me to believe. Another item that would be nice as an IDLE enhancement would be a menu option that applies the fixers (either direction depending on version 2.7 -- 3.2) right in the IDE. Entries that could not be fixed could be flagged for manual update. If there are good tools for app developers to use to make the transition smoother then the development community won't get their ear chewed off so ragged, I'm supposing. kind regards, m harris -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Tuesday, April 5, 2011 2:16:07 AM UTC+1, harrismh777 wrote: Steven D'Aprano wrote: I prefer to consider Python 2.7 and Python 3.x as different dialects of the same language. There are a very few handful of incompatibilities, most of which can be automatically resolved by the 2to3 fixers. Yes, I am actually finding this to be consistent with my experience of trying to come up to speed with 3.2. I have been relieved to find that less has changed than the fear-mongering and bickering was leading me to believe. Another item that would be nice as an IDLE enhancement would be a menu option that applies the fixers (either direction depending on version 2.7 -- 3.2) right in the IDE. Entries that could not be fixed could be flagged for manual update. If there are good tools for app developers to use to make the transition smoother then the development community won't get their ear chewed off so ragged, I'm supposing. Hats off and three cheers to the developers and python community as a whole, as some are down to sugesting easier access to 2-3 rather than OMG! How do I port!! Now that is an excellent sign that Python works. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 4/4/2011 9:16 PM, harrismh777 wrote: Another item that would be nice as an IDLE enhancement would be a menu option that applies the fixers (either direction depending on version 2.7 -- 3.2) right in the IDE. Entries that could not be fixed could be flagged for manual update. I have had the same idea, so naturally I agree ;-). I might even work on it eventually. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Sat, Apr 2, 2011 at 4:01 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: On Fri, 01 Apr 2011 18:22:01 -0700, geremy condra wrote: [...] I don't have a horse in this race, but I do wonder how much of Python could actually survive this test. My first (uneducated) guess is not very much- we would almost certainly lose large pieces of the string API and other builtins, and I have no doubt at all that a really significant chunk of the standard library would vanish as well. In fact, looking at the data I took from PyPI a while back, it's pretty clear that Python's feature set would look very different overall if we applied this test to everything. I don't understand what you mean by this test. I mean testing whether a feature should be in Python based on whether it can meet some undefined standard of popularity if implemented as a third-party module or extension. [...] Granted, but I think the implication is clear: that only those features which could be successful if implemented and distributed by a third party should be in Python. Ah, gotcha. I think you're reading too much into what I said -- I wasn't implying that community support is the only acceptable reason for the existence of features in Python. Development of Python is not a democracy, it is a meritocracy. It is designed by a small team of language developers, starting with Guido van Rossum. Those who do the work decide what goes in, based on whatever combination of factors they choose: I think we're talking at cross purposes. The point I'm making is that there are lots of issues where popularity as a third party module isn't really a viable test for whether a feature is sufficiently awesome to be in core python. As part of determining whether I thought it was appropriate in this case I essentially just asked myself whether any of the really good and necessary parts of Python would fail to be readmitted under similar circumstances, and I think the answer is that very few would come back in. To me, that indicates that this isn't the right way to address this issue, although I admit that I lack any solid proof to base that conclusion on. Geremy Condra -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 3 Apr 2011, at 16:22, geremy condra wrote: I think we're talking at cross purposes. The point I'm making is that there are lots of issues where popularity as a third party module isn't really a viable test for whether a feature is sufficiently awesome to be in core python. As part of determining whether I thought it was appropriate in this case I essentially just asked myself whether any of the really good and necessary parts of Python would fail to be readmitted under similar circumstances, and I think the answer is that very few would come back in. To me, that indicates that this isn't the right way to address this issue, although I admit that I lack any solid proof to base that conclusion on. This has been discussed a few times on python-dev. I think that most developers acknowledge that small-but-high-utility modules would not survive outside of the core because people would simple recreate them rather than investing the time to find, learn and use them. Cheers, Brian -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 3 Apr 2011, at 15:30, harrismh777 wrote: Brian Quinlan wrote: I suspect that this debate is a sink hole that I won't be able to escape from alive but... ... live long and prosper my friend. Something to consider is that OOP philosophy is technically one of the most aesthetic concepts in all of computer science--- with pure functional programming (haskel, erlang) as a close second... You can keep arguing that but you are tilting at windmills. Cheers, Brian -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
geremy condra wrote: Something to consider is that OOP philosophy is technically one of the most aesthetic concepts in all of computer science--- with pure functional programming (haskel, erlang) as a close second... I like how you inserted the word 'technically' in there to give this totally unsubstantiated assertion apparent weight. You picked up on that, did you?:-)) I just solved this little difficulty in my own system ~/bin/ python - /home/mydir/local/python2.7/bin/python2 anaconda - /home/mydir/local/python3.2/bin/python3 ;) kind regards, m harris -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Sun, 03 Apr 2011 16:34:34 +1000, Brian Quinlan wrote: On 3 Apr 2011, at 16:22, geremy condra wrote: I think we're talking at cross purposes. The point I'm making is that there are lots of issues where popularity as a third party module isn't really a viable test for whether a feature is sufficiently awesome to be in core python. As part of determining whether I thought it was appropriate in this case I essentially just asked myself whether any of the really good and necessary parts of Python would fail to be readmitted under similar circumstances, and I think the answer is that very few would come back in. To me, that indicates that this isn't the right way to address this issue, although I admit that I lack any solid proof to base that conclusion on. This has been discussed a few times on python-dev. I think that most developers acknowledge that small-but-high-utility modules would not survive outside of the core because people would simple recreate them rather than investing the time to find, learn and use them. That's certainly true for pure Python code, but for a C extension, the barrier to Do It Yourself will be much higher for most Python coders. On the other hand, for a pure Python function or class, you could stick it on ActiveState's Python cookbook and get some imperfect measure of popularity and/or usefulness from the comments and votes there. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Sun, Apr 3, 2011 at 3:21 AM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: On Sun, 03 Apr 2011 16:34:34 +1000, Brian Quinlan wrote: On 3 Apr 2011, at 16:22, geremy condra wrote: I think we're talking at cross purposes. The point I'm making is that there are lots of issues where popularity as a third party module isn't really a viable test for whether a feature is sufficiently awesome to be in core python. As part of determining whether I thought it was appropriate in this case I essentially just asked myself whether any of the really good and necessary parts of Python would fail to be readmitted under similar circumstances, and I think the answer is that very few would come back in. To me, that indicates that this isn't the right way to address this issue, although I admit that I lack any solid proof to base that conclusion on. This has been discussed a few times on python-dev. I think that most developers acknowledge that small-but-high-utility modules would not survive outside of the core because people would simple recreate them rather than investing the time to find, learn and use them. That's certainly true for pure Python code, but for a C extension, the barrier to Do It Yourself will be much higher for most Python coders. I don't think people will work around it in C. I think they'll grudgingly accept a slow and kludgy python workaround, and more to the point I think they would do that with a vast majority of features at this scale. That's why I say this isn't a good test here- because you could apply it to a great feature or a terrible feature and with overwhelming probability have them fail in both cases. On the other hand, for a pure Python function or class, you could stick it on ActiveState's Python cookbook and get some imperfect measure of popularity and/or usefulness from the comments and votes there. Frankly, I have little trust in this as a measure of popularity. Even PyPI isn't a great indicator, and the numbers you get off of ActiveState are almost certain to be way, way noisier. Geremy Condra -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 4/3/2011 1:26 AM, harrismh777 wrote: Very interesting. Your explanations (and other excellent contributions here) have shown an intense variation of diversity of viewpoint within at least the comp.lang. community with regard to the Python language. If you really want to see variation of opinion, check the archives for discussion of how Python does function call and return. I take a pragmatic viewpoint toward viewpoints: they are tools, so pick ones that work for the particular purpose. My purpose is too make the language easier to learn, especially for those who have not had college CS/programming courses. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Sat, Apr 2, 2011 at 4:17 PM, harrismh777 harrismh...@charter.net wrote: Chris Angelico wrote: Why this lengthy discussion on whether Python is object-oriented or not? What difference does it make? Great question... glad you asked...! But bad things sometimes have to happen. And that's why things are versioned. You didn't read the post... cmp removal is a bad thing, and it does not need to, and should not have to happen. I agree that removal of a feature like that is a bad thing. But the whole point of this thread is the question of whether or not it should have happened (or rather, whether or not it should be undone). Wrong... one of the reasons for OOP in the first place ( OOAD ) is to ensure that BAD THINGS DO NOT HAPPEN. Code reuse and stability are key... and OOAD helps to make sure that works. So it's perfectly acceptable for bad things to happen in a non-object-oriented library, but as soon as it works with objects, it has to eschew badness? This does not make sense. You need to read Grady Booch... and study a little OOP design either using SmallTalk, or C++, to get an appreciation maybe for what is at stake here. I've been a C++ programmer for nearly twenty years. I think I know a few things about OOP. Actually, I've done OOP in non-OO languages; most notably, plain old C. The OS/2 Presentation Manager class hierarchy (SOM) is primarily implemented in C, for instance. My point is that object orientation is completely separate from implementation is separate from interface. You may not have enough knowledge to understand what I'm talking about. Forgive me. Maybe I don't. Let me check my character sheet... Skill, Ranks, Ability, Total modifier Knowledge (Arcana): 19 + 4 = 23 Knowledge (History): 11 + 4 = 15 Knowledge (Nature): 2 + 4 = 6 Knowledge (The Planes): 0 Not sure where Knowledge (OOP) comes in there. Must ask my DM some day. Seriously though: I've programmed various systems, where the division of interface and implementation come in quite different places; my favorite being networking, where the interface is the comms protocol, and everything else is implementation. Objects don't come into it. But this is getting seriously off-topic; none of this connects with the cmp= parameter. Chris Angelico -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Chris Angelico wrote: I've been a C++ programmer for nearly twenty years. I think I know a few things about OOP. Actually, I've done OOP in non-OO languages; most notably, plain old C. The OS/2 Presentation Manager class hierarchy (SOM) is primarily implemented in C, for instance. My point is that object orientation is completely separate from implementation is separate from interface. Thank you for helping me there with your stats... Not sure where Knowledge (OOP) comes in there. Must ask my DM some day. Ok, its an honest question. The answer is the heart of this debate, actually. Please be patient and try to hear this... the bickering on this thread about whether the cmp= removal is a bad thing has been focused on the *wrong issue*. The issue is not whether there is a use-case. The issue is not whether there is a technical reason for justifying the existence of L.sort(cmp= ). The issues debated ad nauseum here by most folks are missing the real point (the main issue). The real issue facing the community in this cmp= debate is whether an established, documented, useful, widely *used* advertised class interface should be removed (don't miss this) for strictly philosophical reasons based on the expectations of the OOP client community in terms of trust and accountability (OOAD) for the established promises of the advertised class interfaces of an *advertised* OOP scripting language--- er, Python. In other words, does the PSF have a responsibility to maintain the L.sort(cmp= key= reverse=) interface for strictly *philosophical* principle based on established norms for *any* OOP language? (and) is there OOAD expectation for this principle? The rest of the thread is arguing for a *technical* determination for inclusion of the cmp= keyword... I am arguing (on the other hand) for a *philosophical* determination for inclusion of the cmp= keyword. But this is getting seriously off-topic; none of this connects with the cmp= parameter. Well, we have to agree to disagree on that point... my view is that this is right-on-topic. Guido should restore the interface, period. And the main point is that this is regardless of technical underpinnings like cruft, performance issues, etc. Now, there may be other issues... CPython to PyPy, for instance, that may or may not affect this ... I don't know. That's not my problem. My problem is that the class list(object).sort(cmp= key= reverse=) interface will be broken in 3.3+ unless somebody argues well for inclusion. Some folks are arguing for inclusion based on technical merit... and that's fine... I am arguing based on philosophical premise and OOAD OOP expectations from the OOP Python client community. Please forgive me, if I made you feel insulted,... that was not intended. Kind regards, m harris -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Fri, 01 Apr 2011 23:54:53 -0500, harrismh777 wrote: It cannot be denied that we are talking exclusively about OOP. End of story. Yes it can be denied. You are categorically *wrong*. Python is a multi- paradigm language that happens to use objects exclusively as its fundamental data type, but even a cursory look at the language proves that it is not exclusively OOP. Python supports: * procedural style programming (like Pascal or Ada or C); * imperative programming (like shell scripts); * functional idioms like lambda, map(), reduce(), list comprehensions (like Haskell or Lisp); * alternatives to classes (procedural based programming again). Important built-ins like len() are not object-oriented; vital commands like import are statement-based rather than OO. All data structures and primitives in Python are objects, but the language is not exclusively object-oriented. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Fri, 01 Apr 2011 15:37:34 -0400, Terry Reedy wrote: On 4/1/2011 3:45 AM, Paul Rubin wrote: What happens then is you define a new interface. In Microsoft-speak if the IWhatever interface needs an incompatible extension like new parameters, they introduce IWhatever2 which supports the new parameters. They change the implementation of IWhatever so it becomes a wrapper for IWhatever2 setting the new parameters to default values, to keep implementing the old behavior. This is exactly the sort of cruft I am talking about. Languages can survive a small amount of cruft, but eventually they bloat to the point that the only thing keeping them going is inertia. They become dinosaurs, like COBOL or PL\I, or bloated monstrosities like .Net, Java or C++ that nobody *likes* but merely keep using because they have to. No offense to anyone who actually likes .Net, C++ or Java *wink* If we follow this rule (more of a guideline really) of no interface changes ever, eventually we will be up to Python 5.2 and IWhatever12. This imposes serious maintenance costs, documentation costs, learning costs for new users, and even decision costs when you write a function: Should I use the list, list2, sortable_list, sortable_list2, sortable_lost3, [note spelling, which we're stuck with forever], heterogeneous_list, heterogeneous_list_without_stooge_sort, new_list, fancy_list, fancy_list2, fancy_list_with_extra_oomph, newer_than_new_list or list3? Each and every interface carries a cost. Even if it is small, that cost must be weighed up against the benefits, rather than declaring that all interfaces are sacred once published. Removing a published interface imposes a one-time cost on those using that interface, but it has an on-going benefit for all. Now you have two versions, and eventually many more, to maintain and document. That takes resources we currently do not have. Exactly. The no interface changes ever rule assumes that the human resources dedicated to maintenance are unlimited, that the cost of keeping multiple interfaces around is zero, and that the cost of removing an interface is infinite. None of those assumptions are even *remotely* true in the real world. (If the cost of removal were assumed to be merely large, or even huge, then there could be circumstances where the benefit outweighed the cost, and the rule would be *Almost* never change a published interface.) To be sure, interface stability is a good thing. But it is not the only good thing, and it is not so good that it always outweighs every other consideration. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Fri, 01 Apr 2011 18:22:01 -0700, geremy condra wrote: [...] I don't have a horse in this race, but I do wonder how much of Python could actually survive this test. My first (uneducated) guess is not very much- we would almost certainly lose large pieces of the string API and other builtins, and I have no doubt at all that a really significant chunk of the standard library would vanish as well. In fact, looking at the data I took from PyPI a while back, it's pretty clear that Python's feature set would look very different overall if we applied this test to everything. I don't understand what you mean by this test. I mean testing whether a feature should be in Python based on whether it can meet some undefined standard of popularity if implemented as a third-party module or extension. [...] Granted, but I think the implication is clear: that only those features which could be successful if implemented and distributed by a third party should be in Python. Ah, gotcha. I think you're reading too much into what I said -- I wasn't implying that community support is the only acceptable reason for the existence of features in Python. Development of Python is not a democracy, it is a meritocracy. It is designed by a small team of language developers, starting with Guido van Rossum. Those who do the work decide what goes in, based on whatever combination of factors they choose: * some features are such obvious no-brainers that only a complete idiot would leave them out (what do you mean, there's no way to add two numbers?); * what other languages do; * personal preference; * tools that they personally find useful, or that they expect will be useful to many; etc. And *every one of these* is subject to the requirement of a rough consensus, or a BDFL pronouncement. The rest of us can only hope to persuade the Python developers: if you want somebody to scratch your itch instead of their own, you need to convince them to do so. My point was that good community support is a fairly good method of persuasion. The broader community does not get a vote, but that does not mean their voices are unheard. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
I suspect that this debate is a sink hole that I won't be able to escape from alive but... On 2 Apr 2011, at 19:29, harrismh777 wrote: In other words, does the PSF have a responsibility to maintain the L.sort(cmp= key= reverse=) interface for strictly *philosophical* principle based on established norms for *any* OOP language? (and) is there OOAD expectation for this principle? No, there should be no expectation that Python 2.x interfaces be preserved in Python 3.x unless they have demonstrated utility. Furthermore, there should be no expectation that a particular interface survive for more than a few major Python versions. PEP-004 describes how deprecations are expected to proceed at module granularity. The rest of the thread is arguing for a *technical* determination for inclusion of the cmp= keyword... I am arguing (on the other hand) for a *philosophical* determination for inclusion of the cmp= keyword. Any argument along what you call philosophical grounds will not be successful. Technical (including aesthetic, convenience, etc.) arguments *may* be successful. Cheers, Brian -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Fri, 01 Apr 2011 19:29:59 -0700, Paul Rubin wrote: Steven D'Aprano steve+comp.lang.pyt...@pearwood.info writes: What I'm saying is this: cmp is already removed from sorting, and we can't change the past. Regardless of whether this was a mistake or not, No it's not already removed, I just tried it (in Python 2.6, which is called Python for short) and it still works. It's not removed from Python until basically all Python users have migrated and Python essentially always means Python 3. You know full well that I'm talking about Python 3, not Python 2 or Python 1. In Python 2, sort still takes a cmp argument, so what's the problem? Until that happens, for Python 2 users, Python 3 is just a fork of Python with some stuff added and some stuff broken, that might get its act together someday. I see in the subject of this thread, Guido rethinking removal of cmp from sort method which gives hope that one particular bit of breakage might get fixed. You call it a breakage, but many others disagree. The point of this thread was supposed to be to encourage people like you to come up with good, reasoned, reasonable arguments for adding cmp, not to engage in FUD about Python 3 being a fork of Python, or that there is never under any circumstances any good reason for removing features. I expected a certain amount of bitterness to come through, but if I had realised just what a bunch of whining I was going to unleash, I would have kept quiet. [...] If sorting is in the stdlib like functools is, then the similarity makes sense and the suggestion isn't so bad. But you're proposing a 3rd party module, which is not the same thing at all. In the face of opposition from senior developers who don't want cmp in the language, and who will presumably oppose adding a needless module to the standard library, you will need good solid evidence that this functionality is wanted and needed, and not just a bunch of crappy rationalizations like I can't be bothered thinking up a key function, which was actually suggested by someone in this thread. (Not in those exact words, but that's the gist of it.) A good way to gather such evidence is to make it a third party module first: if people want the feature, they will install it, just like they install numpy or pyparsing or any other wanted module that is not in the standard library. And if not, in the absolute worst case, at least *you* can continue using cmp in your own code. I realise that this strategy will only imperfectly capture community desire for cmp sorting, but do you have a better strategy? You don't have to follow my suggestion. If you think you have a better strategy for convincing the people doing the actual work to scratch *your* itch instead of their own, then go right ahead and use it. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 4/2/2011 4:29 AM, harrismh777 wrote: I am responding to both this and a previous post of yours. Python is not a thing, but an abstraction of multiple parts. It is a name, a trademark of the Python Software Foundation. It is a Platonic ideal in the mind of Guido and others. It is a series* of versions intended to approach that ideal. That goal requires pruning as well as additions. * with three subseries: 0.9 to 2.1, 2.2 to 2.7, 3.0 to 3.x I call it object-based because in the plain English meaning of the words, that is what it is and has been from the start. It is a language for manipulating information objects that have identity, value, and (type, type or class, and class, respectively, in the three subseries). Python is object-based as opposed to memory-block-based, like C and other language. Understanding the difference is important for understanding or using Python. [I do not care if before or, more likely, after Guido started Python, some other people defined Object-Based Language in a way that excludes Python. That does not give them ownership of 'object' and 'based'.] I do not call Python object-oriented for two reasons. (And I am not responsible for what anyone else says.) 1. Python started as purely type-based with no classes (in this respect, like C). So if OOP means classes with inheritance, OOP does not apply to all of Python. It only became completely class-based in 3.0 when the obsolete Class and Instance *types* were omitted. They are still present in 2.7. [The interfaces of these types were and are, of course, published. It is ironic that a removal that you philosophically oppose makes your desired designation of Python as object-oriented more plausible ;-).] 2. Calling it object-oriented somehow induces some people to fantasize that Python 'should' conform to a particular, non-Pythonic idea of OOP and that developers 'should' or 'have a responsibility' to conform to said person's ideas. In other words, does the PSF have a responsibility to maintain the L.sort(cmp= key= reverse=) interface for strictly *philosophical* principle based on established norms for *any* OOP language? No. I say this based on the philosophical principle that obligations are reciprocal. Each x.y version (and indeed, each x.y.z release) is a gift from the developers to the world, available to anyone who wants it. Gifts do not confer obligations on the giver, even if one thinks there might be some on the recipient. In any case, cmp= is still present and has not been removed from any version in which it ever appeared. It was not retained in 3.0 because Guido thought that omitting it would move 3.0 closer to ideal Python. Python 3 was *always* intended to be something of a break from 2.x. I believe I was one of the first to use the term, during the discussion (8 years ago or so) of changing the meaning of int/int (which is to say, the meaning of int.__div__(a,b)). I suggested that the first version in which the old meaning was dropped should be called 3.0 to *signify that there would be some breakage of 2.x code*. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Steven D'Aprano wrote: It cannot be denied that we are talking exclusively about OOP. End of story. Yes it can be denied. All data structures and primitives in Python are objects, but the language is not exclusively object-oriented. Yeah, I know, Steven. The discussion, from which my quote was pulled from context, was in response to whether Python should be viewed as object-based or object-oriented. Terry says object-based, my view is object-oriented. (the reasons for both have already been stated) I can use C++ for procedural programming (and I often do, to take advantage of the //comments and iostreams cin and cout). But even though I use C++ for procedural programming, I still know at heart that its an object-oriented language---and a good thing too, or else there wouldn't be an iostreams class to take advantage of. :) I view Python the same way--- it is object-oriented and has some obligation to the OOAD paradigm even though it can be used procedurally and|or functionally. Of course my functional experimentation and research resides almost exclusively with haskel and erlang, I have dabbled with Python's lambda and have enjoyed playing a bit with the functional aspects of Python... albeit, I still consider Python an object oriented language at heart. I am gaining an understanding for the rich diversity of viewpoints within this community regarding the evolution of this fantastic language. I had no idea the viewpoints were *so* diverse. kind regards, m harris -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Terry Reedy wrote: In other words, does the PSF have a responsibility to maintain the L.sort(cmp= key= reverse=) interface for strictly *philosophical* principle based on established norms for *any* OOP language? No. I say this based on the philosophical principle that obligations are reciprocal. In any case, cmp= is still present and has not been removed from any version in which it ever appeared. It was not retained in 3.0 because Guido thought that omitting it would move 3.0 closer to ideal Python. Python 3 was *always* intended to be something of a break from 2.x. Very interesting. Your explanations (and other excellent contributions here) have shown an intense variation of diversity of viewpoint within at least the comp.lang. community with regard to the Python language. Always learning is lifelong... Except for the long tradition for an eternal object reference to SPAM SPAM SPAM and Monte Python's Flying Circus, perhaps this new pythonesque language might be called Anaconda? ... going with the python serpent image on the covers of some python books. It does seem that there is a sort-of-drive, as it were, towards some ideal concept that is maintained in an orderly state of flux while being practically useful and without too much disruption... for which I laud the entire team. Well, winding down the debate side of this thing from my standpoint, if Guido does not restore the interface along the lines of my *philosophical* ideal, then I cannot see why he would consider restoring it at all. Clearly, his own arguments for removing the cmp= are solid, well thought out, and make sense from the standpoint technically for moving towards Anaconda. The new Python will be lean, clean, and more pythonesque than ever before. All is good. Thanks for the great discussion. For those of you still clam-mering for inclusion of the cmp= keyword, you have my most sincere wishes, encouragement, and remorse. :) kind regards, m harris -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Brian Quinlan wrote: I suspect that this debate is a sink hole that I won't be able to escape from alive but... ... live long and prosper my friend. Something to consider is that OOP philosophy is technically one of the most aesthetic concepts in all of computer science--- with pure functional programming (haskel, erlang) as a close second... :) kind regards, m harris -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Sat, Apr 2, 2011 at 10:30 PM, harrismh777 harrismh...@charter.net wrote: Brian Quinlan wrote: I suspect that this debate is a sink hole that I won't be able to escape from alive but... ... live long and prosper my friend. Something to consider is that OOP philosophy is technically one of the most aesthetic concepts in all of computer science--- with pure functional programming (haskel, erlang) as a close second... I like how you inserted the word 'technically' in there to give this totally unsubstantiated assertion apparent weight. Geremy Condra -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Steven D'Aprano wrote: The difference between implementation and interface is not specific to object-oriented code -- When I speak of implementation vs interface I am speaking from a strictly object oriented philosophy, as pedigree, from Grady Booch, whom I consider to be the father of Object Oriented Analysis and Design (Booch, OO AD with apps, 1994). He makes this remarkable statement: The interface of a class is the one place where we assert all of the assumptions that a client may make about any instances [objects] of the class; the implementation encapsulates details about which no client may make assumptions (Booch, p50, 1994). The Class interface holds a special firmness which fosters the client relationship of trust and assumption, without which OOAD is pointless. The interface of a Class must not change once advertised, and once in production. This is specific to OOAD. Encapsulation of the abstractions must reside within the implementation about which no one may make assumptions. This is also where all of the cruft (if any) resides. Cruft never resides at the interface, only in the encapsulation of the Class abstractions, and only if the Class was poorly designed. To put my advice in MY other words, here are my two rules about Class interface based on Booch, but in terms that are mine, and which are way simpler to put into practice: Rule 1: Never change an advertised Class interface. Corollary 1)a The Class interface never encapsulates cruft. Corollary 1)b The Class implementation usually encapsulates cruft. Corollary 1)c Only the Class implementation may be changed. Rule 2: If an advertised Class interface must be changed to remove cruft, please re-read Rule #1, and review corollaries 1)a, 1)b, and 1)c. As an aside, but in response to your straw man argument regarding obscure cruft, none of the bickering regarding this cmp issue is about obscure code. We are discussing one of the primary methods of one of the primary Class objects are the lovely heart of Python--- the list.sort(). This is NOT obscure. Guido arbitrarily altered the interface to one of the MOST important Class objects in all of the Python language. This is not good, and it does violate the spirit of OOAD. If the interface changes had been left to type promotion on integer divide, and the print() method, then folks would probably not be in such an uproar over 3x. But Guido over reached the bounds of credulity by altering a Class method that *many* people find useful, desirable, efficacious and easy... they use it and the love it (right or wrong). The client(s) are making assumptions about the List Class interface and they have an OOAD right to do so... and they are left with their rear-ends handing out in the breeze because their assumptions are false. Not good. Kind regards, m harris -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Fri, Apr 1, 2011 at 5:13 PM, harrismh777 harrismh...@charter.net wrote: Corollary 1)a The Class interface never encapsulates cruft. Provably false. Even a well-designed interface, if it is asked to deal with a changing implementation and changing requirements, will eventually either acquire cruft, or be found too rigid to be of use. The only interface immune to this is the transparent interface (effectively: just package up the parameters in a single string and pass it through), which isn't really an interface at all, it just exposes something else. In everything else, cruft is a reality that must be dealt with. Chris Angelico -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Terry Reedy wrote: Python 3 was announced and as a mildly code breaking version at least 5 years before it came out. I appreciate the spirit of your arguments overall, and I do not necessarily disagree with much of what you are saying. I would like to challenge you to see this from a little different perspective, if I may. There are two distinct ways for looking at this mild code breakage, and it might be good to think about how we approach changes in the future based on an understanding of both perspectives, and consideration for the clients. In the possible perspective of the Python language developers 3x changes are mild (in fact, overall, probably even insignificant percentage-wise). Ok, we removed the cmp comparison keyword from list.sort(), made the print() method consistent with the rest of the language, and correctly promoted integer divide 1/2 to float so that the answer is something greater than zero! Fine. Looks like just a couple little changes, no big deal, stop your whining. The perspective of the Class client is something quite different. They do not look at the overall percentage of Python language definition that has changed (tiny percentage, right) they look at the overall percentage of their own projects that have just broken and need to be rewritten to accommodate the disruption in the advertised Class interface. And to the client-- these tiny changes are magna-bodacious! (nobbled thinking) You guys have changed integer divide---! the PRINT print() functionality is diff e r e n t ---! and for crying out loud you changed S O R T ( ) !!! I wonder if folks like google need to do in-place sorts over lists very often ...? I wonder how mnay Python scripts call the list.sort() method with the cmp keyword specified... ? (could it be thousands, or millions?) All the hoorah you guys are getting, as well all of this incessant bickering over cmp, is some of the answer to these questions. When you get ready to change an advertised Class interface in the future, please consider my interface rules (I gave them to Steven) and please take into account your client base---the ones who are making valid assumptions about your Class interfaces. Its easy, and its most considerate. king regards, m harris -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Chris Angelico ros...@gmail.com writes: Provably false. Even a well-designed interface, if it is asked to deal with a changing implementation and changing requirements, will eventually either acquire cruft, or be found too rigid to be of use. What happens then is you define a new interface. In Microsoft-speak if the IWhatever interface needs an incompatible extension like new parameters, they introduce IWhatever2 which supports the new parameters. They change the implementation of IWhatever so it becomes a wrapper for IWhatever2 setting the new parameters to default values, to keep implementing the old behavior. Removing cmp certainly isn't the most disruptive change of Python 3, but it seems like the one with the least benefit. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 4/1/2011 3:45 AM, Paul Rubin wrote: Removing cmp certainly isn't the most disruptive change of Python 3, That was almost certainly the ascii to unicode switch for strings. It is still not quite complete in 3.2 but should be pretty well ironed out in 3.3. but it seems like the one with the least benefit. Since the set of changes was finite, there *must* be (at least) one with the lowest benefit/cost ratio. The removal of list.sort(cmp) may well be that one. Certainly, reasonable people can disagree as to whether the ratio is above or below 1.0. If cmp had not been removed, some other change would have been the worst in this respect, and possibly the subject of a thread like this. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 4/1/2011 2:13 AM, harrismh777 wrote: When I speak of implementation vs interface I am speaking from a strictly object oriented philosophy, as pedigree, from Grady Booch, whom I consider to be the father of Object Oriented Analysis and Design (Booch, OO AD with apps, 1994). Python is object based but not object oriented in the Booch sense. The Class interface holds a special firmness which fosters the client relationship of trust and assumption, without which OOAD is pointless. The interface of a Class must not change once advertised, and once in production. This is specific to OOAD. Right, and Python is not OOAD based. Never change an advertised Class interface. In Python, class interfaces are no more sacred than module or function interfaces. If one takes 'no interface change' literally, then Python would have to be frozen. Even bug fixes change a defacto interface. If you want frozon, stick with one particular release. There are lots available. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 4/1/2011 3:45 AM, Paul Rubin wrote: What happens then is you define a new interface. Like key= versus cmp= In Microsoft-speak if the IWhatever interface needs an incompatible extension like new parameters, they introduce IWhatever2 which supports the new parameters. They change the implementation of IWhatever so it becomes a wrapper for IWhatever2 setting the new parameters to default values, to keep implementing the old behavior. If cmp had been left (or were reinstated) but its implementation changed internally to use cmp_to_key to make it a wrapper for key, you would be happy? 2to3 could probably gain a fixer to change .sort(cmp=f) # to import functools import cmp_to_key .sort(key=functools.cmp_to_key(f)) I know some would not like this because interface change is not their real concern. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 4/1/2011 2:44 AM, harrismh777 wrote: Terry Reedy wrote: Python 3 was announced and as a mildly code breaking version at least 5 years before it came out. I appreciate the spirit of your arguments overall, and I do not necessarily disagree with much of what you are saying. I would like to challenge you to see this from a little different perspective, if I may. I have, but I consider your perspective, as I understand it, unrealistic There are two distinct ways for looking at this mild code breakage, and it might be good to think about how we approach changes in the future based on an understanding of both perspectives, and consideration for the clients. Guido especially and the other developers, generally, try to balance benefit and cost. What some may not know is that we consider benefits over several years, and benefits to future users as well as current users. We hope and expect the former to outnumber the latter in not too many years. The decision, about the time of 2.2, to put off most non-bugfix code-breaking changes until 3.0, rather than spread them over the remainder of 2.x, was in large part based on the expressed wishes of at least some users. (I myself would have preferred sooner and more spread out.) In the possible perspective of the Python language developers 3x changes are mild Compared to the changes to both the language and implementation Guido once considered, they are! But the Perl 6 fiasco and an article by Joel Spolsky advocating evolutionary, not revolutionary, software change prompted him toward mininal change that would meet objectives. Many proposed changes were rejected. The perspective of the Class client is something quite different. There is no 'Class' in Python. Module and function clients have the same perspective. Python classes are just instances of class 'type'. Modules, instances of class 'module', are sometimes regarded as singleton classes. Functions are instances of various classes. Methods are functions until bound as instances of bound-method classes. Functions are sometimes an alternative to classes, one favored by those of a more functional rather than strictly OOP bent. In any case, you seem to include the interface of class attributes (such as the list.sort function) as part of the class interface. Since anything can be a class attribute, freezing 'class interfaces' freezes everything. When you get ready to change an advertised Class interface in the future, please consider my interface rules Taken strictly, they pretty much prohibit anything but implementation changes. If we allow the addition of numbered versions of modules, classes, and functions, with nothing ever deleted, then we would have massive bloat, with attendant maintenance and documentation problems. Some modules might be up to v20 by now. Certainly, the switch from ascii to unicode as the default text encoding (and I am not sure your rules would allow that) changed the interface of nearly every module and a large fraction of classes. Let me reiterate that PSF and Python are not Microsoft and Office. Resources are primarily voluntary and limited, but code and even binaries are available indefinitely, so no one is forced by *us* to switch any particular Python program to any other version of the language. If you think your rules are workable, try developing a fork that obeys them but wait, maybe there already is one: 2.7, which will only get bugfixes and invisible implementation changes for several years. Of course, implementation changes must be screened carefully because in the real world, as opposed to the imagined Booch world, it is all to0 easy to accidentally introduce an interface change for some corner case. Also, an implementation change that exchanges space for time will be seen an an interface change by somebody. Do you include space-time behavior in *your* definition of interface (that should not be changed)? Indeed, some object to the removal of cmp precisely on this basis, not on the fairly trivial code rewrite it entails. This was the case with the Google example that prompted this thread. The Googler may well have written fresh code, as code breakage was *not* the issue. If the list.sort change had been purely an implementation change, if cmp remained but its had been changed to use cmp_to_key internally, there would have been many of the same objections expressed on this thread anyway. So why is there a problem with cmp? Because there are people who want most of the changes that break your rules, but not this particular one. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Fri, Apr 1, 2011 at 10:06 AM, Terry Reedy tjre...@udel.edu wrote: On 4/1/2011 2:13 AM, harrismh777 wrote: When I speak of implementation vs interface I am speaking from a strictly object oriented philosophy, as pedigree, from Grady Booch, whom I consider to be the father of Object Oriented Analysis and Design (Booch, OO AD with apps, 1994). Python is object based but not object oriented in the Booch sense. Actually, IINM: 1) C++, Java, and Python are object oriented in common vernacular and the Booch sense. 2) Ada and Modula-2 are object based in common vernacular and the Booch sense. References: http://en.wikipedia.org/wiki/Object-based http://www.dotnetspider.com/forum/153678-Object-Based-Vs-Object-oriented-language.aspx -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Terry Reedy tjre...@udel.edu writes: What happens then is you define a new interface. Like key= versus cmp= Well, in an untyped language like Python, adding a feature to an interface doesn't require defining a new interface unless you change something incompatibly. key= is very useful but it can be added without breaking cmp= . 2to3 could probably gain a fixer to change .sort(cmp=f) # to import functools import cmp_to_key .sort(key=functools.cmp_to_key(f)) I know some would not like this because interface change is not their real concern. Looks like a good idea. There is an efficiency hit from the above in some situations, but at least it prevents code from breaking, so unless there's some drawback I'm not currently spotting, it's better than nothing and I'd endorse adding such a wrapper. 2to3 should show some kind of diagnostic and maybe put a comment into the output code, when it does that particular transformation, since most of the time there's probably a better way to write the key function. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Terry Reedy tjre...@udel.edu writes: Never change an advertised Class interface. In Python, class interfaces are no more sacred than module or function interfaces. If one takes 'no interface change' literally, then Python would have to be frozen. Even bug fixes change a defacto interface. Oh come on, a interface is advertised if it's documented in the manual, which cmp is. There are some undocumented (what you call defacto) interfaces that you sometimes have to be pragmatically a bit careful about messing with because people rely on them, but it's almost always more legitimate to break an undocumented interface than a documented one. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 4/1/2011 3:45 AM, Paul Rubin wrote: What happens then is you define a new interface. In Microsoft-speak if the IWhatever interface needs an incompatible extension like new parameters, they introduce IWhatever2 which supports the new parameters. They change the implementation of IWhatever so it becomes a wrapper for IWhatever2 setting the new parameters to default values, to keep implementing the old behavior. Now you have two versions, and eventually many more, to maintain and document. That takes resources we currently do not have. Some problems in addition to the benefits of this approach: 1. The users of IWhatever will not gain the benefits of IWhatever2. 2. Upgrading to IWhatever2 requires a change of name as well as off parameters. 3. If only some users are upgraded, the IWhatever and IWhatever2 users may become incompatible even though they were before, thus breaking code without changing the old interface. Example: Python2 added str2 (= unicode) on top of str. This had all the problems listed above. Since CPython uses str internally, in particular for identifiers, CPython users were stuck with the limitations of str. Rebinding str to mean str2 has many benefits, especially in the future, in addition to current costs. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Terry Reedy tjre...@udel.edu writes: But the Perl 6 fiasco Perl 6 a complete failure? Wow, must be coming from a clueless Python zealot. If Perl 6 is a fiasco, so is Python 3. Both are not considered production ready, and both can be downloaded and used today: http://rakudo.org/ Next release is planned for later this month. Did Perl 6 take a long time? Sure. But confusing it with Python 2 - Python 3 is just plainly stupid. It's a complete rewrite of Perl, and it's much easier to think of it as a complete new language instead of similar to Perl 4 - 5 and comparable to Python 2 - 3. But if you had any idea what you were talking about, you already knew that. -- John Bokma j3b Blog: http://johnbokma.com/Facebook: http://www.facebook.com/j.j.j.bokma Freelance Perl Python Development: http://castleamber.com/ -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Mon, Mar 28, 2011 at 10:11 PM, Dan Stromberg drsali...@gmail.com wrote: On Mon, Mar 28, 2011 at 6:58 PM, Paul Rubin no.email@nospam.invalidwrote: DSU is a clever and useful design pattern, but comparison sorting is what all the sorting textbooks are written about. Actually, even though I wrote one program that could almost benefit from cmp sorting that might have trouble with key sorting, I think it was probably the right decision to remove cmp sorting. I must (partially) recant this. It's not clear to me this was a good decision. Key sorting gives a nice performance boost (because it frequently allows a sort to be run without any calls to python code), but it's not a complete replacement for cmp sorting because cmp sorting does not require exposing implementation detail (at least, not without some foresight and extra typing) to a module's client. I like fast, but not so much that I think we should make abstractions impure. I don't care that much about modifying code across 2.x to 3.x, but modifying client code now and then in 3.x because of a misfeature - that bothers me. If we must have just one, cmp sorting is probably better - just for code beauty. I'm not sure having both would be a bad thing really. They seem distinct enough from one another, as well as complementary enough, that IMO it'd be justified. It strikes me as there's more than one way to do it only if you define it rather broadly. On the other hand... What if the key function were to become something other than a callable? Like a tuple of (list.use_cmp_fn, cmp_fn)? It's ugly, but it'd allow you to have the fast sorting normally, and fall back on something more general when the need surfaces - without requiring a change in client code. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Wed, Mar 30, 2011 at 7:13 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: snip Or, an alternative approach would be for one of the cmp-supporters to take the code for Python's sort routine, and implement your own sort-with- cmp (in C, of course, a pure Python solution will likely be unusable) and offer it as a download. For anyone who knows how to do C extensions, this shouldn't be hard: just grab the code in Python 2.7 and make it a stand- alone function that can be imported. If you get lots of community interest in this, that is a good sign that the solution is useful and practical, and then you can push to have it included in the standard library or even as a built-in. And if not, well, at least you will be able to continue using cmp in your own code. I don't have a horse in this race, but I do wonder how much of Python could actually survive this test. My first (uneducated) guess is not very much- we would almost certainly lose large pieces of the string API and other builtins, and I have no doubt at all that a really significant chunk of the standard library would vanish as well. In fact, looking at the data I took from PyPI a while back, it's pretty clear that Python's feature set would look very different overall if we applied this test to everything. Geremy Condra -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Terry Reedy tjre...@udel.edu writes: IWhatever2 setting the new parameters to default values Now you have two versions, and eventually many more, to maintain and document. In the case of cmp= there's not two interfaces needed. Python2 does a perfectly good job supporting cmp and key with one interface. We do have urllib and urllib2 as separate interfaces (with separate implementations), and we keep some other legacy interfaces around too, like the sha and md5 modules (both of which are now subsumed by hashlib). That takes resources we currently do not have. Well ok, but now you're making excuses for instability, which seems like a problem given Python's self-marketing as a stable, production-class system that competes with better-funded efforts like Java. Example: Python2 added str2 (= unicode) on top of str. This had all the problems listed above. Since CPython uses str internally, in particular for identifiers, CPython users were stuck with the limitations of str. Rebinding str to mean str2 has many benefits, especially in the future, in addition to current costs. That really is a massive change, but a welcome one, because Python2 programs broke all the time due to missed conversions between str and unicode. It is the type of thing that a language transition (Python2 to Python3) is supposed to bring, that goes beyond Interface2 to Interface3. There's been a lot of experience gained in the decades since Python's creation and it's fine to do an overhaul after all these years. The issues are 1) don't break stuff unless there's a substantial benefit; and 2) don't do these major incompatible releases too often. There should not be any incompatible Python4 before 2020 or so. I actually think Python3 actually didn't go far enough in fixing Python2. I'd have frankly preferred delaying it by a few years, to allow PyPy to come to maturity and serve as the new main Python implementation, and have that drive the language change decisions. Instead we're going to have to give up a lot of possible improvements we could have gotten from the new implementation. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Fri, 01 Apr 2011 14:31:09 -0700, geremy condra wrote: On Wed, Mar 30, 2011 at 7:13 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: snip Or, an alternative approach would be for one of the cmp-supporters to take the code for Python's sort routine, and implement your own sort-with- cmp (in C, of course, a pure Python solution will likely be unusable) and offer it as a download. For anyone who knows how to do C extensions, this shouldn't be hard: just grab the code in Python 2.7 and make it a stand- alone function that can be imported. If you get lots of community interest in this, that is a good sign that the solution is useful and practical, and then you can push to have it included in the standard library or even as a built-in. And if not, well, at least you will be able to continue using cmp in your own code. I don't have a horse in this race, but I do wonder how much of Python could actually survive this test. My first (uneducated) guess is not very much- we would almost certainly lose large pieces of the string API and other builtins, and I have no doubt at all that a really significant chunk of the standard library would vanish as well. In fact, looking at the data I took from PyPI a while back, it's pretty clear that Python's feature set would look very different overall if we applied this test to everything. I don't understand what you mean by this test. I'm certainly not suggesting that we strip every built-in of all methods and make everything a third-party C extension. That would be insane. Nor do I mean that every feature in the standard library should be forced to prove itself or be removed. The features removed from Python 3 were deliberately few and conservative, and it was a one-off change (at least until Python 4000 in the indefinite future). If something is in Python 3 *now*, you can assume that it won't be removed any time soon. What I'm saying is this: cmp is already removed from sorting, and we can't change the past. Regardless of whether this was a mistake or not, the fact is that it is gone, and therefore re-adding it is a new feature request. Those who want cmp functionality in Python 3 have three broad choices: (1) suck it up and give up the fight; the battle is lost, move on; (2) keep arguing until they either wear down the Python developers or get kill-filed; never give up, never surrender; (3) port the feature that they want into a third-party module, so that they can actually use it in code, and then when they have evidence that the community needs and/or wants this feature, then try to have it re- added to the language. I'm suggesting that #3 is a more practical, useful approach than writing another hundred thousand words complaining about what a terrible mistake it was. Having to do: from sorting import csort as a prerequisite for using a comparison function is not an onerous requirement for developers. If fans of functional programming can live with from functools import reduce, fans of cmp can live with that. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Fri, 01 Apr 2011 14:37:20 -0700, Paul Rubin wrote: I actually think Python3 actually didn't go far enough in fixing Python2. I'd have frankly preferred delaying it by a few years, to allow PyPy to come to maturity and serve as the new main Python implementation, and have that drive the language change decisions. Instead we're going to have to give up a lot of possible improvements we could have gotten from the new implementation. There's always Python 4000 :) -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Fri, 01 Apr 2011 13:42:11 -0600, John Bokma wrote: Terry Reedy tjre...@udel.edu writes: But the Perl 6 fiasco Perl 6 a complete failure? Fiasco does not mean complete failure. It is a debacle, an embarrassing, serious failure, (and also an Italian wine bottle with a rounded bottom), but not necessarily complete. It does not imply that a fiasco cannot, with great effort and/or time, be eventually recovered from. Netscape Navigator 6 was a fiasco, which directly lead to the dominance of Internet Explorer in the browser market, but today the heir of Netscape, Mozilla's Firefox browser, has regained a majority of the browser market in Europe and is catching up on IE world-wide. Those who are old enough will remember that Microsoft Word 3.0 was a fiasco, but there's no doubt that Word has well recovered to virtually own the entire word processing market. Wow, must be coming from a clueless Python zealot. Thanks for sharing. If Perl 6 is a fiasco, so is Python 3. Both are not considered production ready, and both can be downloaded and used today: This is FUD about Python 3. Python 3 is absolutely production ready. The only reason to avoid Python 3 is if your software relies on a specific third-party library that does not yet support Python 3. On the other hand, the PerlFAQ still describes Perl 6 as not ready: http://faq.perl.org/perlfaq1.html#What_are_Perl_4_Perl http://faq.perl.org/perlfaq1.html#What_is_Perl_6_ Perl 6 is the next major version of Perl, although it's not intended to replace Perl 5. It's still in development in both its syntax and design. The work started in 2002 and is still ongoing. ... Perl 6 is not scheduled for release yet ... Nine years after Perl 6 was started, neither the syntax nor design are settled. The initial PEP for Python 3000 development was in 2006; the first final release of Python 3 was in 2008, but I don't count that because Python 3.0 was fatally flawed and is no longer supported. The first production ready release of Python 3.1 was 2009: three years from talking to a production-ready version. Did Perl 6 take a long time? Sure. But confusing it with Python 2 - Python 3 is just plainly stupid. It's a complete rewrite of Perl, and it's much easier to think of it as a complete new language instead of similar to Perl 4 - 5 and comparable to Python 2 - 3. What you have described is not a reason for rejecting the claim that Perl 6 was a fiasco, but the reason for *why* it was a fiasco. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Fri, Apr 1, 2011 at 5:41 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: On Fri, 01 Apr 2011 14:31:09 -0700, geremy condra wrote: On Wed, Mar 30, 2011 at 7:13 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: snip Or, an alternative approach would be for one of the cmp-supporters to take the code for Python's sort routine, and implement your own sort-with- cmp (in C, of course, a pure Python solution will likely be unusable) and offer it as a download. For anyone who knows how to do C extensions, this shouldn't be hard: just grab the code in Python 2.7 and make it a stand- alone function that can be imported. If you get lots of community interest in this, that is a good sign that the solution is useful and practical, and then you can push to have it included in the standard library or even as a built-in. And if not, well, at least you will be able to continue using cmp in your own code. I don't have a horse in this race, but I do wonder how much of Python could actually survive this test. My first (uneducated) guess is not very much- we would almost certainly lose large pieces of the string API and other builtins, and I have no doubt at all that a really significant chunk of the standard library would vanish as well. In fact, looking at the data I took from PyPI a while back, it's pretty clear that Python's feature set would look very different overall if we applied this test to everything. I don't understand what you mean by this test. I mean testing whether a feature should be in Python based on whether it can meet some undefined standard of popularity if implemented as a third-party module or extension. I'm certainly not suggesting that we strip every built-in of all methods and make everything a third-party C extension. That would be insane. Granted, but I think the implication is clear: that only those features which could be successful if implemented and distributed by a third party should be in Python. My argument is that there are many features currently in Python that I doubt would pass that test, but which should probably be in anyway. The conclusion I draw from that is that this isn't a particularly good way to determine whether something should be in standard Python. Nor do I mean that every feature in the standard library should be forced to prove itself or be removed. The features removed from Python 3 were deliberately few and conservative, and it was a one-off change (at least until Python 4000 in the indefinite future). If something is in Python 3 *now*, you can assume that it won't be removed any time soon. I may have been unclear, so let me reiterate: I'm not under the impression that you're advocating this as a course of action. I'm just pointing out that the standard for inclusion you're advocating is probably not a particularly good one, especially in this case, and engaging in a bit of a thought experiment about what would happen if other parts of Python were similarly scrutinized. What I'm saying is this: cmp is already removed from sorting, and we can't change the past. Regardless of whether this was a mistake or not, the fact is that it is gone, and therefore re-adding it is a new feature request. Those who want cmp functionality in Python 3 have three broad choices: I might quibble over whether re-adding is the same as a new feature request, but as I said- I don't care about cmp. (1) suck it up and give up the fight; the battle is lost, move on; (2) keep arguing until they either wear down the Python developers or get kill-filed; never give up, never surrender; (3) port the feature that they want into a third-party module, so that they can actually use it in code, and then when they have evidence that the community needs and/or wants this feature, then try to have it re- added to the language. I'm suggesting that #3 is a more practical, useful approach than writing another hundred thousand words complaining about what a terrible mistake it was. Having to do: from sorting import csort as a prerequisite for using a comparison function is not an onerous requirement for developers. If fans of functional programming can live with from functools import reduce, fans of cmp can live with that. And that's fine, as I said I don't have a horse in this race. My point is just that I don't think the standard you're using is a good one- ISTM that if it *had* been applied evenly we would have wound up with a much less complete (and much less awesome) Python than we have today. That indicates that there are a reasonable number of real-world cases where it hasn't and shouldn't apply. Geremy Condra -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 4/1/2011 3:22 PM, Paul Rubin wrote: 2to3 could probably gain a fixer to change .sort(cmp=f) # to import functools import cmp_to_key .sort(key=functools.cmp_to_key(f)) I know some would not like this because interface change is not their real concern. Looks like a good idea. There is an efficiency hit from the above in some situations, but at least it prevents code from breaking, so unless there's some drawback I'm not currently spotting, it's better than nothing and I'd endorse adding such a wrapper. 2to3 should show some kind of diagnostic and maybe put a comment into the output code, when it does that particular transformation, since most of the time there's probably a better way to write the key function. rewriting cmp_to_key in C is underway http://bugs.python.org/issue11707 -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Steven D'Aprano steve+comp.lang.pyt...@pearwood.info writes: What I'm saying is this: cmp is already removed from sorting, and we can't change the past. Regardless of whether this was a mistake or not, No it's not already removed, I just tried it (in Python 2.6, which is called Python for short) and it still works. It's not removed from Python until basically all Python users have migrated and Python essentially always means Python 3. Until that happens, for Python 2 users, Python 3 is just a fork of Python with some stuff added and some stuff broken, that might get its act together someday. I see in the subject of this thread, Guido rethinking removal of cmp from sort method which gives hope that one particular bit of breakage might get fixed. the fact is that it is gone, and therefore re-adding it is a new feature request. Those who want cmp functionality in Python 3 have three broad choices: ... (3) port the feature that they want into a third-party module, ... I'm suggesting that #3 is a more practical, useful approach ... Having to do: from sorting import csort ... If fans of functional programming can live with from functools import reduce, fans of cmp can live with that. If sorting is in the stdlib like functools is, then the similarity makes sense and the suggestion isn't so bad. But you're proposing a 3rd party module, which is not the same thing at all. Batteries included actually means something, namely that you don't have to write your critical applications using a library base written with a Wikipedia-like development model where anybody can ship anything, where you're expected to examine every module yourself before you can trust it. Stuff in the stdlib occasionally has bugs or gaps, but it has a generally consistent quality level, is documented, and has been reviewed and reasonably sanity checked by a central development group that knows what it's doing. Stuff in 3rd party libraries has none of the above. There are too many places for it to go wrong and I've generally found it best to stick with stdlib modules instead of occasionally superior modules that have the disadvantage of coming from a third party. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Steven D'Aprano steve+comp.lang.pyt...@pearwood.info writes: There's always Python 4000 :) Is that on the boards yet? -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Sat, Apr 2, 2011 at 1:29 PM, Paul Rubin no.email@nospam.invalid wrote: If sorting is in the stdlib like functools is, then the similarity makes sense and the suggestion isn't so bad. But you're proposing a 3rd party module, which is not the same thing at all. Batteries included actually means something... To me, batteries included means that I can: 1) Write a Python script on any Ubuntu laptop that I put my hands on, and expect it to work. 2) Put a shebang on it, chmod it plus exx, and give it to someone, and expect it to work on his system. 3) Post it to my web site along with the comment You will need a Python interpreter to run this, and expect it to work. Every third-party library I need weakens that. Sure, situation 1 isn't too hard; but the other two end up becoming a bit awkward. The Yosemite Project requires a support module on Windows, making it that bit harder to share with people; but I accept that, because it's doing some rather unusual things (simulating keypresses on another window). Sorting a list is not unusual enough to justify a third-party module. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Paul Rubin no.email at nospam.invalid writes: I actually think Python3 actually didn't go far enough in fixing Python2. I'd have frankly preferred delaying it by a few years, to allow PyPy to come to maturity and serve as the new main Python implementation, and have that drive the language change decisions. Instead we're going to have to give up a lot of possible improvements we could have gotten from the new implementation. Why would having PyPy as the reference implementation have made this design decisions turn out better? -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Benjamin Peterson benja...@python.org writes: Why would having PyPy as the reference implementation have made this design decisions turn out better? A fair amount of Python 2's design was influenced by what was convenient or efficient to implement in CPython. There's nothing wrong with that and it's a perfectly normal and sensible strategy. Anyone writing Python code in a serious way has to maintain some awareness of how CPython works, so CPython's influence finds its way into Python user programs too. With PyPy as the reference implementation, the designers may find they can take the language in cool new directions that were impossible with CPython, or alternatively, they might find that adding minor retrictions (that would count as breaking more stuff) would give big advantages under PyPy that weren't significant in CPython. What kinds of stuff and is any of it a sure thing? Unknown. That's why the idea was: first get more experience with PyPy, then figure out how it should affect the language. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Fri, Apr 1, 2011 at 8:43 PM, Paul Rubin no.email@nospam.invalid wrote: Benjamin Peterson benja...@python.org writes: Why would having PyPy as the reference implementation have made this design decisions turn out better? A fair amount of Python 2's design was influenced by what was convenient or efficient to implement in CPython. It's dangerous for a language to be defined by a single implementation for too long. Also, a language defined by an implementation in a flexible language is more likely to escape some problems than a language defined by an implementation in a rigid language. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Terry Reedy wrote: When I speak of implementation vs interface I am speaking from a strictly object oriented philosophy, as pedigree, from Grady Booch, whom I consider to be the father of Object Oriented Analysis and Design (Booch, OO AD with apps, 1994). Python is object based but not object oriented in the Booch sense. . . . and Python is not OOAD based. With due respects Terry, these statements you have made are categorically not true. Even a casual perusal of the Python supplied documentation and helps (documented class interfaces) make it very clear that OOP and OOAD are at the very heart of the design of Python. To help clear this up a bit, and keeping in line with the cmp notion of this thread, I have posted (at the very end of this entry) a portion of the class definition for list(object) from several releases, each of which resides on at least one of my production servers: 2.3 2.4.1 2.5.2 2.6.2 2.7.1 3.2 I have snipped out the center section of the class definition but have left the heading and the sort() method for comparison. I am going to try to be making several points with these examples as follows: 1) cmp is integral to advertised class interface since (2005). 2) OOP and OOAD are integral to Python definition, period. 3) semantics aside, removing cmp breaks the spirit of OOAD First, all of the primary books tout (even sport) Python as object oriented (not just object based). This is important to note; just a couple of examples: [Python] is commonly defined as an object-oriented scripting language--a definition that blends support for OOP with an overall orientation toward scripting roles (Lutz, Learning Python, 4th ed, 2009). We use the term base class to refer to a class that is inherited; a base class may be the immediate ancestor, or may be further up the inheritance tree (Summerfield, Programming in Python 3: A complete Introduction to the Python Language, 2nd ed, 2010). It cannot be denied that we are talking exclusively about OOP. End of story. Granted, Python certainly does not implement the Booch OOAD like SmallTalk does, nor Eiffel, nor C++. Regardless, it is very clear that the OOAD concepts from Booch are present consciously or unconsciously in most of what Python is doing. To really get a good feel for this all one has to do is use the help() method from the Python interpreter command prompt and then enter list at the help prompt. I have attached at the end of this item the class list(object) definitions for several Python versions showing the defined Methods :--- specifically the sort(). First it will be clearly seen that we are looking at an OOP OOAD class definition with internal and external attributes and methods, defined as a class with function methods, including sort(). To say that Python is not based in OOAD is to make an asinine statement. Very obviously OOAD was at the lovely heart of the Python design. Obviously Booch didn't get credit for it..., but his methodology is very clearly present. Second it can be clearly seen that beginning in 2.3 there was needed a comparison function. In the Programming Python book 3rd ed the comparison worked with the class definition like this: L.sort( lambda x, y,: cmp(x['n'], y['n']) ) (Lutz, Programming Python, 3rd ed, 2006) In 2.4 cmpfunc= was replaced with (cmp= key= reverse=) which remains with the definition through 2.7.1 today. It is clear that the cmp= keyword is not in any way cruft, nor is it in any way obscure. It was needed, wanted, useful, desired, and most importantly implemented and advertised. End of the sad sad story. The Python clients (yes, I'm one of those) must be able to make assumptions about the class definitions of an advertised object-oriented language. Otherwise, the excellent OOAD work that obviously went into the class list(object).sort() is pointless. If the clients using sort() cannot rely on the advertised class interface for their projects into the future then their use of the language is weakened, and maybe in the end not as useful. Python is obviously OOP with the concepts of OOAD at the heart of the project's design. Denying this truth in order to justify breaking the spirit of OOAD is not only not helpful, it adds insult to injury. But this is worse, because programmers are not the only ones who rely on the advertised interface. When the interface is in flux the only people who benefit short term are the language designers. The clients who might otherwise benefit long term may never see that day because they are forced to remain on a previous release far into the future. (@Steven ---this is not FUD... No Fear, Absolutely Certain, No Doubt) But the other folks who lose on this type of nobbled thinking are the ones who write the books, and the tutorials, and the ones who teach the students... and of course
Re: Guido rethinking removal of cmp from sort method
On Sat, Apr 2, 2011 at 3:54 PM, harrismh777 harrismh...@charter.net wrote: Terry Reedy wrote: When I speak of implementation vs interface I am speaking from a strictly object oriented philosophy, as pedigree, from Grady Booch, whom I consider to be the father of Object Oriented Analysis and Design (Booch, OO AD with apps, 1994). Python is object based but not object oriented in the Booch sense. . . . and Python is not OOAD based. With due respects Terry, these statements you have made are categorically not true. Even a casual perusal of the Python supplied documentation and helps (documented class interfaces) make it very clear that OOP and OOAD are at the very heart of the design of Python. Why this lengthy discussion on whether Python is object-oriented or not? What difference does it make? The implementation vs interface difference has nothing to do with that. If I write a library, and I export a number of public functions, then updating the library in a way that breaks programs is a bad thing. But bad things sometimes have to happen. And that's why things are versioned. Object oriented programming/design/analysis/architecture/whatever has no effect on that; if you link against a library, or call on an object, you need to be able to depend on it. The interface of sort([1,2,3,4]) is no different from [1,2,3,4].sort() just because one is objects and the other is functions. Maybe we need a new directive: from __past__ import cmpsort Chris Angelico -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Terry Reedy wrote: So why is there a problem with cmp? Because there are people who want most of the changes that break your rules, but not this particular one. Sadly, Terry, there may be some truth in there. Often folks whine about this or that, and they have absolutely no reason (or konwledge) upon which to base their whining, and often the rants are based in petty jealousy or abstract foolishness. Please, don't let those morons keep you from seeing my point of view, nor from trying to understand where some of your reasonable clients are basing their opposition. I'm a scholar, a software design engineer (IBM for twenty-five years), and a joyful hacker with enough playful smarts to at least be entertaining. Bottom line, I care. kind regards, m harris -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Chris Angelico wrote: Why this lengthy discussion on whether Python is object-oriented or not? What difference does it make? Great question... glad you asked...! But bad things sometimes have to happen. And that's why things are versioned. You didn't read the post... cmp removal is a bad thing, and it does not need to, and should not have to happen. Object oriented programming/design/analysis/architecture/whatever has no effect on that; Wrong... one of the reasons for OOP in the first place ( OOAD ) is to ensure that BAD THINGS DO NOT HAPPEN. Code reuse and stability are key... and OOAD helps to make sure that works. if you link against a library, or call on an object, you need to be able to depend on it. The interface of sort([1,2,3,4]) is no different from [1,2,3,4].sort() just because one is objects and the other is functions. You need to read Grady Booch... and study a little OOP design either using SmallTalk, or C++, to get an appreciation maybe for what is at stake here. Yes, Python OOP is optional for the client perspective. But if chosen, OOP must WORK as expected by the OOP community if the language is touted as being able to support OOP, which also implies support for the spirit of OOAD. You may not have enough knowledge to understand what I'm talking about. Forgive me. kind regards, m harris -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Antoon Pardon wrote: On Sun, Mar 13, 2011 at 12:59:55PM +, Steven D'Aprano wrote: The removal of cmp from the sort method of lists is probably the most disliked change in Python 3. On the python-dev mailing list at the moment, Guido is considering whether or not it was a mistake. If anyone has any use-cases for sorting with a comparison function that either can't be written using a key function, or that perform really badly when done so, this would be a good time to speak up. How about a list of tuples where you want them sorted first item in ascending order en second item in descending order. Greetings, Not sure here, but thought you folks might want to have another viewpoint from someone who is maybe not so close to the trees as to be able to comment effectively on the forest. Many of you (Guido included) have lost significant sight of a critical object oriented philosophical pillar (but not all of you, thank goodness). To cut right to the heart of it--- NEVER change an advertised interface. Change the implementation to your little hearts desire, but never alter an advertised interface (code is based on it, books are based on it, tutorials are based on it, college comp sci courses are based on it... etc). If cmp parm is utilized and is helpful or perceived as useful and existing code requires it then for crying out loud leave it alone. I think your overall worship of Guido as Python King has left many of you with nobbled thinking. On the other hand, this debate and constant bickering is a serious rehash of an ancient discussion without any new points--- irritating. Take a look at issue 1771 http://bugs.python.org/issue1771 particularly msg #s 95975 and 95982 but no, read the whole thing it is very clear that Guido wants to restructure the python language for consistency and elegance (and who can blame him?). He makes some very good arguments for the justification of removing the cmp keyword from list.sort() and builtin.sorted()/ but, that is not the point... he is breaking a fundamental law of object oriented programming... don't break and advertised interface (particularly if it is useful and people are actually making use of it!). This is insane folks. Python is one of the most elegant OO languages to gain popular appeal and wide-spread use. This sad broken move from 2.x to 3.x has the potential of ruining the language for everyone. Many of us dropped JAVA (compile once debug everywhere) because it is complicated, a pita to use, slow, and actually doesn't port too well on any platform... and here we go again with python. How do the developers of python ever expect folks to be able to make use of the language into the future when every time we turn around the interface is broken or being debated in flux? Many of us want to use the new 3.2+ version, but no one is going to ship it pre-installed (probably for many years) because of this issue. There is no way to easily migrate, nor backport, and everyone is going to be forced to develop code (and have to maintain code) on two distinct branches (for many years to come). I suspect that people are just going to stick with back versions for a long time. We need to get a grip on this, people. Guido does not need a use case; he just needs to restore the interface that everyone expects. Put up a message to start to use key= instead, and plan deprecation for something like five years out... this just seems like such a no-brainer to me. But then, maybe its just that I'm too inexperienced to know any better. sigh In the mean time... I'm playing around with 3.2 and really liking it... hoping that the world will actually be using it someday. Kind regards, m harris -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Thu, 31 Mar 2011 01:34:17 -0500, harrismh777 wrote: Many of you (Guido included) have lost significant sight of a critical object oriented philosophical pillar (but not all of you, thank goodness). To cut right to the heart of it--- NEVER change an advertised interface. Thanks for your comments, M Harris, but I'm afraid you stumble right at the first step. The difference between implementation and interface is not specific to object-oriented code -- it is no more, or less, an object oriented philosophical pillar than it is a pillar of functional programming, procedural programming, or any other programming paradigm (except perhaps spaghetti coding). Nor should you say NEVER. To put your advice in other words: Once you have added a feature, no matter how badly thought out, broken, rarely used, or even actively hated by the entire programming community, you must continue supporting it forever, adding more and more cruft around it, just in case some obscure piece of software that hasn't been maintained for ten years relies on it. Nobody has forgotten the principle of separating interface from implementation. This was not an accidental backwards incompatible change, it was a deliberate decision made in the full knowledge that it would be a painful move for some, but also in the expectation that *not* removing cruft becomes even more painful over time. Cruft has real costs: maintenance costs, development costs, runtime costs, learning costs, and usability costs. Any language which does not prune cruft from time to time will ossify and is doing its users a real disservice. Just not too often. Nobody is denying that the loss of list.sort(cmp=...) will hurt some people, or some use-cases. The disagreement is over the question of whether the pain from its loss outweighs the utility of its removal. Python is about halfway through the expected migration period from 2.x to 3.x, and so far it has not been worse or more difficult than expected. Support for Python 3 is about what was expected (probably better than expected), uptake of Python 3 by users is about what was expected, the number of nay-sayers, Negative Nellies and trolls claiming that Python 3 will destroy the language is about what was expected, and the number of credible forks of the language promising to support Python 2 forever is exactly what was expected: zero. [...] Many of us dropped JAVA (compile once debug everywhere) because it is complicated, a pita to use, slow, and actually doesn't port too well on any platform... Perhaps that's because Java needs to drop some cruft. Many of us want to use the new 3.2+ version, but no one is going to ship it pre-installed (probably for many years) because of this issue. Wrong. There is at least one Linux distro, Arch, which ships with Python3 as the system Python. Arch is pretty bleeding edge, but where they go, others will follow. Besides, pre-installed loses a lot of importance when installation is as easy as sudo apt-get python3. There is no way to easily migrate, nor backport, and everyone is going to be forced to develop code (and have to maintain code) on two distinct branches (for many years to come). Your FUD is outdated. There are now many major products which support Python 2 and 3, and some of them out of the same code base. Here is a small selection: http://allanmcrae.com/2010/12/python-3-module-support/ http://mail.mems-exchange.org/durusmail/qp/441/ http://nedbatchelder.com/blog/200910/running_the_same_code_on_python_2x_and_3x.html Note that last link is from 2009. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Thu, Mar 31, 2011 at 02:13:53AM +, Steven D'Aprano wrote:
On Wed, 30 Mar 2011 11:06:20 +0200, Antoon Pardon wrote:
As far as I can see, key will only produce significant speedups, if
comparing items can then be completly done internally in the python
engine without referencing user python code.
Incorrect. You don't even need megabytes of data to see significant
differences. How about a mere 1000 short strings?
[steve@wow-wow ~]$ python2.6
Python 2.6.6 (r266:84292, Dec 21 2010, 18:12:50)
[GCC 4.1.2 20070925 (Red Hat 4.1.2-27)] on linux2
Type help, copyright, credits or license for more information.
from random import shuffle
data = ['a'*n for n in range(1000)]
shuffle(data)
from timeit import Timer
t_key = Timer('sorted(data, key=lambda a: len(a))',
... 'from __main__ import data')
t_cmp = Timer('sorted(data, cmp=lambda a,b: cmp(len(a), len(b)))',
... 'from __main__ import data')
min(t_key.repeat(number=1000, repeat=5))
0.89357517051696777
min(t_cmp.repeat(number=1000, repeat=5))
7.6032949066162109
That's almost ten times slower.
But how does it contradict what I wrote above? Maybe I didn't make
myself clear but in your example, the key function produces ints.
With ints the comparison is completely done within the python engine
without the need to defer to user python code (through the rich
comparison functions). So this is the kind of key I described that
would produce a significant speedup.
But once your key produces user objects that need to be compared
through user python code with __lt__ methods, getting a speedup
through the use of key instead of cmp will not be obvious and
in a number of case you will loose speed.
Of course, the right way to do that specific sort is:
t_len = Timer('sorted(data, key=len)', 'from __main__ import data')
min(t_len.repeat(number=1000, repeat=5))
0.64559602737426758
which is even easier and faster. But even comparing a pure Python key
function to the cmp function, it's obvious that cmp is nearly always
slower.
I don't find that obvious at all.
Frankly, trying to argue that cmp is faster, or nearly as fast, is a
losing proposition. In my opinion, the only strategy that has even a
faint glimmer of hope is to find a convincing use-case where speed does
not matter.
I don't argue such a general statement. I just argue there are cases
where it is and I supporeted that statement with numbers. The only
way these numbers could be disputed was by focussing on specific details
that didn't generalize.
Now maybe providing a cmp_to_key function written in C, can reduce
the overhead for such cases as Raymond Hettinger suggested. If it
turns out that that works out, then I would consider this thread
usefull even if that will be the only result of this thread.
Something else the dev team can consider, is a Negation class.
This class would wrap itself around a class or object but reverse the ordering.
So that we would get Negation('a') Negation('b'). That would make it
easy to create sort keys in which partial keys had to be sorted differently
from each other. This would have to be done by the dev team since I
guess that writing such a thing in Python would loose all the speed
of using a key-function.
Or, an alternative approach would be for one of the cmp-supporters to
take the code for Python's sort routine, and implement your own sort-with-
cmp (in C, of course, a pure Python solution will likely be unusable) and
offer it as a download. For anyone who knows how to do C extensions, this
shouldn't be hard: just grab the code in Python 2.7 and make it a stand-
alone function that can be imported.
If you get lots of community interest in this, that is a good sign that
the solution is useful and practical, and then you can push to have it
included in the standard library or even as a built-in.
And if not, well, at least you will be able to continue using cmp in your
own code.
I'll first let Raymond Hettinger rewrite cmp_to_key in C, as I understand he
suggested, and reevaluate after that.
--
Antoon Pardon
--
http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Antoon Pardon antoon.par...@rece.vub.ac.be writes: Something else the dev team can consider, is a Negation class This would have to be done by the dev team since I guess that writing such a thing in Python would loose all the speed of using a key-function. That is a good idea. SQL has something like it, I think, and Haskell also has it. I thought about writing it in Python but it would slow things down a lot. I hadn't thought of writing it in C for some reason. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 3/31/2011 2:34 AM, harrismh777 wrote: breaking a fundamental law of object oriented programming... don't break and advertised interface (particularly if it is useful and people are actually making use of it!). This is insane folks. Each x.y version (starting with 2.3) is feature stable: just bug fixes, no additions (except as occasionally required to fix a bug), no removals. 2.x had few feature changes or removals, as it was planned and announced that all the big ones were being delayed to 3.x. 2.7 will be polished for years, depending on how long anyone is willing to work on it. I have been a bit surprised and disappointed that no 2.x fan has yet come forward (that I know of) to specifically help fix 2.7 bugs. Guido does not need a use case; he just needs to restore the interface that everyone expects. Put up a message to start to use key= instead, and plan deprecation for something like five years out... this just Python 3 was announced and as a mildly code breaking version at least 5 years before it came out. Old-style classes were clearly doomed when the new object. The int division change was also decided on about that time. I agree that it would have been better if the developer group had been large enough to make the transition more orderly, but ... as it is, it was not until 2.7 was out that we took a really good look at library interfaces and found that 'public interface' was not as well defined as it should be for some modules. That is much clearer for many in 3.2. In the mean time... I'm playing around with 3.2 and really liking it... Good. hoping that the world will actually be using it someday. I already am and fully expect it to be the dominant version in not too many years. Some linux distributions have already made it the default. Others have and will hold back. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Tue, Mar 29, 2011 at 03:35:40PM -0400, Terry Reedy wrote: For anyone interested, the tracker discussion on removing cmp is at http://bugs.python.org/issue1771 There may have been more on the old py3k list and pydev list. One point made there is that removing cmp= made list.sort consistent with all the other comparision functions, min/max/nsmallest/nlargest/groupby that only have a key arg. How many would really want cmp= added everywhere? I wouldn't have a problem with it. I would also like to react to the following. Guido van Rossum in msg95975 on http://bugs.python.org/issue1771 wrote: | Also, for all of you asking for cmp back, I hope you realize that | sorting N values using a custom cmp function makes about N log N calls | calls to cmp, whereas using a custom key calls the key function only N | times. This means that even if your cmp function is faster than the | best key function you can write, the advantage is lost as N increases | (which is just where you'd like it to matter most :-). This is a play on semantics. If you need python code to compare two items, then this code will be called N log N times, independently of the fact how this code is presented, as a cmp function or as rich comparison methods. So forcing people to write a key function in cases where this will only result in the cmp code being translated to __lt__ code, accomplishes nothing. As far as I can see, key will only produce significant speedups, if comparing items can then be completly done internally in the python engine without referencing user python code. A minor problem problem with cmp is that the mapping between return values and input comparisons is somewhat arbitrary. Does -1 mean ab or ba? (That can be learned and memorized, of course, though I tend to forget without constant use). My rule of thumb is that a b is equivallent with cmp(a, b) 0 A bigger problem is that it conflicts with key=. What is the result of l=[1,3,2] l.sort(cmp=lambda x,y:y-x, key=lambda x: x) print l ? (for answer, see http://bugs.python.org/issue11712 ) While that can also be learned, I consider conflicting parameters undesireable and better avoided when reasonably possible. So I see this thread as a discussion of the meaning of 'reasonably' in this particular case. But what does this have to do with use cases? Does what is reasonable depend on the current use cases without regard of possible future use cases? Is the conflict between key and cmp a lesser problem in the case of someone having a huge data set to sort on a computer that lacks the resources to decorate as opposed to currently noone having such a data set? Are we going to decide which functions/methods get a cmp argument depening on which use cases we currently have that would need it? This thread started with a request for use cases. But if you take this kind of things into consideration, I don't see how use cases can then make a big difference in the final decision. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Wed, 30 Mar 2011 11:06:20 +0200, Antoon Pardon wrote:
As far as I can see, key will only produce significant speedups, if
comparing items can then be completly done internally in the python
engine without referencing user python code.
Incorrect. You don't even need megabytes of data to see significant
differences. How about a mere 1000 short strings?
[steve@wow-wow ~]$ python2.6
Python 2.6.6 (r266:84292, Dec 21 2010, 18:12:50)
[GCC 4.1.2 20070925 (Red Hat 4.1.2-27)] on linux2
Type help, copyright, credits or license for more information.
from random import shuffle
data = ['a'*n for n in range(1000)]
shuffle(data)
from timeit import Timer
t_key = Timer('sorted(data, key=lambda a: len(a))',
... 'from __main__ import data')
t_cmp = Timer('sorted(data, cmp=lambda a,b: cmp(len(a), len(b)))',
... 'from __main__ import data')
min(t_key.repeat(number=1000, repeat=5))
0.89357517051696777
min(t_cmp.repeat(number=1000, repeat=5))
7.6032949066162109
That's almost ten times slower.
Of course, the right way to do that specific sort is:
t_len = Timer('sorted(data, key=len)', 'from __main__ import data')
min(t_len.repeat(number=1000, repeat=5))
0.64559602737426758
which is even easier and faster. But even comparing a pure Python key
function to the cmp function, it's obvious that cmp is nearly always
slower.
Frankly, trying to argue that cmp is faster, or nearly as fast, is a
losing proposition. In my opinion, the only strategy that has even a
faint glimmer of hope is to find a convincing use-case where speed does
not matter.
Or, an alternative approach would be for one of the cmp-supporters to
take the code for Python's sort routine, and implement your own sort-with-
cmp (in C, of course, a pure Python solution will likely be unusable) and
offer it as a download. For anyone who knows how to do C extensions, this
shouldn't be hard: just grab the code in Python 2.7 and make it a stand-
alone function that can be imported.
If you get lots of community interest in this, that is a good sign that
the solution is useful and practical, and then you can push to have it
included in the standard library or even as a built-in.
And if not, well, at least you will be able to continue using cmp in your
own code.
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Mon, Mar 28, 2011 at 11:35:09AM +, Steven D'Aprano wrote: [...] Forcing people to use a key-function, will produce cases where python will ask for more memory and take longer to sort, than allowing to provide a cmp function. More memory yes; take longer to sort, almost certainly not (except, perhaps, for a very narrow window where the addition of a key causes the application to page out to virtual memory, but a comparison function wouldn't -- and even then, the key function might still be faster). But we've already covered the more memory angle to death. We already understand that. This is why I have asked if there are any *other* use- cases for a comparison function. Dan Stromberg has suggested that another use-case would be lazy- comparisons, where you might not be able to generate a single key cheaply, but you can perform a comparison lazily. This for the simple reason that for some order-functions, the only way to write a key-function will be to write a specific class, that will implement the order with the same kind of code that otherwise would have been put in the cmp function, There is a recipe for that on ActiveState's website, google for cmp_to_key. That recipe has been added to Python's standard library for version 3.2. Please stop flogging that dead horse. This doesn't make any sense. As far as I can see there is nothing in Dan's code that would make it more troublesome to use with cmp_to_key than Carl Bank's examples or that would make it loose more speed than the overhead I was describing. Whatever is meant by lazily, the cmp_to_key function will generate a key function that during the sorting will finally defer to this cmp-function to do the comparisons. If the cmp is lazy when it does its work as a cmp-argument, it will be lazy when it does its work when a key is wrapped around it. This argument works for whatever property, the cmp-function should posses. If your cmp argument has property foo when comparing during a sort then using cmp_to_key will preserve than property when comparing during a sort. This for the simple reason that with a cmp_to_key function when the sort comapares two keys, it will after some overhead just call the cmp function with the undecorated data. That means that asking for use cases that fall outside the overhead in memory or speed of the cmp_to_key function and are not about the cumbersome work in writing your own key-function in order to reduce that overhead, that asking for such use cases is like asking for a real number that squares to a negative. We can conclude a priori that there won't be any. That is why I don't consider a request for such use cases a serious request. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Mon, Mar 28, 2011 at 03:43:03PM +, Steven D'Aprano wrote: On Mon, 28 Mar 2011 14:39:04 +0200, Antoon Pardon wrote: I tried to sort lists of 1 elemens. Each element a tuple two items that was to be sorted first according to the first item in ascending order, then according to the second item in descending order. This was done on python 2.6 so I had to write my own cmp_to_key function. I then sorted these lists of 1 elements a 1000 times with the following methods. Thank you for spending the time to get some hard data, but I can't replicate your results since you haven't shown your code. Rather than attempt to guess what you did and duplicate it, I instead came up with my own timing measurements. Results are shown here, my code follows below: But why expect me to come up with such data? Shouldn't the dev team have come up with such data before they decided to remove the cmp-argument, instead of afterwards expecting the python users to come up with data to support the reversing of that decision? [steve@sylar ~]$ python2.7 sort_test.py Comparison function: 12.1256039143 Key function: 3.51603388786 Double sort: 2.33165812492 cmp_to_key: 28.1129128933 By far the best result comes from two calls to sort. Not far off is the key function solution. (Admittedly, coming up with a key function for non- numeric data would be challenging.) The comparison function, and the cmp_to_key solution, are clearly *much* slower. So, that means that the fastest solutions don't generalize. The double sort is useless if the actual sorting is done in a different module/function/method than the module/function/method where the order is implemented. It is even possible you didn't write the module where the sorting actually occurs. Your key-function only works with numeric data, strings already pose a problem here. The cmp function can be externally provided, without you having the source. So sooner or later there will be a user who just doesn't have a choice but to use cmp_to_key or something similar. and who will have to suffer this kind of speed loss. -- Antoon Pardon -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Steven D'Aprano wrote: On Fri, 25 Mar 2011 10:21:35 +0100, Antoon Pardon wrote: On Thu, Mar 24, 2011 at 11:49:53PM +, Steven D'Aprano wrote: On Thu, 24 Mar 2011 17:47:05 +0100, Antoon Pardon wrote: However since that seems to be a problem for you I will be more detailed. The original poster didn't ask for cases in which cmp was necessary, he asked for cases in which not using cmp was cumbersome. I'm the original poster, and that's not what I said. I said: If anyone has any use-cases for sorting with a comparison function that either can't be written using a key function, or that perform really badly when done so, this would be a good time to speak up. You'll notice that I said nothing about whether writing the code was easy or cumbersome, and nothing about readability. Well fine. I should have realised the question was just a pretense and that there really never was any intention to consider the reactions, because the answer is already fixed. Of course a key function can always be written, it may just need a specific class to implement the specific order. Likewise there is no reason to expect the order-functions to preform worse when implemented in a class, rather than in a function. The reason Guido is considering re-introducing cmp is that somebody at Google approached him with a use-case where a key-based sort did not work. The use-case was that the user had masses of data, too much data for the added overhead of Decorate-Sort-Undecorate (which is what key does), but didn't care if it took a day or two to sort. So there is at least one use-case for preferring slowly sorting with a comparison function over key-based sorting. I asked if there any others. It seems not. That was known from the begining, was it ? Since key sort trades memory for CPU, you could expect that masses of data use cases may fail because of memory shortage. JM -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Tue, Mar 29, 2011 at 1:46 AM, Antoon Pardon antoon.par...@rece.vub.ac.be wrote: The double sort is useless if the actual sorting is done in a different module/function/method than the module/function/method where the order is implemented. It is even possible you didn't write the module where the sorting actually occurs. There's a good point in the above. Usually the order in which a class needs to be sorted, is a detail that should be hidden by the class (or auxiliary comparison function). Doing more than one sort in order to get one key forward and another in reverse, seems to require a client of the class to know class detail (or auxiliary comparison function detail). In fact, if something inside the class (or auxiliary comparison function) needs to change s.t. you must change how you sort its instances, then you might have to change a bunch of single-sorts to multiple-sorts in client code. EG if you were sorting a number in reverse followed by a string forward, to a number in reverse followed by a string forward and another string in reverse. In principle, you could come up with a negate string operation though. EG for Latin-1, each ch would become chr(255-ord(ch)). That's probably going to be rather expensive, albeit linear. It should at least avoid the need for multi-sorting and exposing implementation detail. I want to say you could do something similar for Unicode strings, but I don't have enough experience with Unicode to be sure yet. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 29/03/2011 18:01, Dan Stromberg wrote: On Tue, Mar 29, 2011 at 1:46 AM, Antoon Pardon antoon.par...@rece.vub.ac.be mailto:antoon.par...@rece.vub.ac.be wrote: The double sort is useless if the actual sorting is done in a different module/function/method than the module/function/method where the order is implemented. It is even possible you didn't write the module where the sorting actually occurs. There's a good point in the above. Usually the order in which a class needs to be sorted, is a detail that should be hidden by the class (or auxiliary comparison function). Doing more than one sort in order to get one key forward and another in reverse, seems to require a client of the class to know class detail (or auxiliary comparison function detail). In fact, if something inside the class (or auxiliary comparison function) needs to change s.t. you must change how you sort its instances, then you might have to change a bunch of single-sorts to multiple-sorts in client code. EG if you were sorting a number in reverse followed by a string forward, to a number in reverse followed by a string forward and another string in reverse. In principle, you could come up with a negate string operation though. EG for Latin-1, each ch would become chr(255-ord(ch)). That's probably going to be rather expensive, albeit linear. It should at least avoid the need for multi-sorting and exposing implementation detail. I want to say you could do something similar for Unicode strings, but I don't have enough experience with Unicode to be sure yet. You would have to do more than that. For example, A, but if you negate both strings you get \xBE, not \xBE. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Wed, Mar 30, 2011 at 5:57 AM, MRAB pyt...@mrabarnett.plus.com wrote: You would have to do more than that. For example, A, but if you negate both strings you get \xBE, not \xBE. Strings effectively have an implicit character at the end that's less than any other character. Easy fix: Append a character that's greater than any other. So A becomes \xFF \xBE\xFF. Still not going to be particularly efficient. Chris Angelico -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
For anyone interested, the tracker discussion on removing cmp is at http://bugs.python.org/issue1771 There may have been more on the old py3k list and pydev list. One point made there is that removing cmp= made list.sort consistent with all the other comparision functions, min/max/nsmallest/nlargest/groupby that only have a key arg. How many would really want cmp= added everywhere? A minor problem problem with cmp is that the mapping between return values and input comparisons is somewhat arbitrary. Does -1 mean ab or ba? (That can be learned and memorized, of course, though I tend to forget without constant use). A bigger problem is that it conflicts with key=. What is the result of l=[1,3,2] l.sort(cmp=lambda x,y:y-x, key=lambda x: x) print l ? (for answer, see http://bugs.python.org/issue11712 ) While that can also be learned, I consider conflicting parameters undesireable and better avoided when reasonably possible. So I see this thread as a discussion of the meaning of 'reasonably' in this particular case. -- Terry Jan Reedy -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Tue, Mar 29, 2011 at 1:08 PM, Chris Angelico ros...@gmail.com wrote: On Wed, Mar 30, 2011 at 5:57 AM, MRAB pyt...@mrabarnett.plus.com wrote: You would have to do more than that. For example, A, but if you negate both strings you get \xBE, not \xBE. Strings effectively have an implicit character at the end that's less than any other character. Easy fix: Append a character that's greater than any other. So A becomes \xFF \xBE\xFF. Still not going to be particularly efficient. Not to mention that it still has bugs: \0 \xff \xff\xff -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Wed, Mar 30, 2011 at 6:48 AM, Ian Kelly ian.g.ke...@gmail.com wrote: Not to mention that it still has bugs: \0 \xff \xff\xff That's because \0 isn't less than any character, nor is \xff greater than any. However, the latter is true if (for instance) your string is in UTF-8; it may be possible to use some construct that suits your own data. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Tue, Mar 29, 2011 at 12:48 PM, Ian Kelly ian.g.ke...@gmail.com wrote:
On Tue, Mar 29, 2011 at 1:08 PM, Chris Angelico ros...@gmail.com wrote:
On Wed, Mar 30, 2011 at 5:57 AM, MRAB pyt...@mrabarnett.plus.com
wrote:
You would have to do more than that.
For example, A, but if you negate both strings you get
\xBE, not \xBE.
Strings effectively have an implicit character at the end that's less
than any other character. Easy fix: Append a character that's greater
than any other. So A becomes \xFF \xBE\xFF.
Still not going to be particularly efficient.
Not to mention that it still has bugs:
\0
\xff \xff\xff
--
http://mail.python.org/mailman/listinfo/python-list
It probably could be a list of tuples:
'abc' - [ (1, chr(255 - ord('a')), (1, chr(255 - ord('b')), (1, chr(255 -
ord('c')), (0, '') ]
...where the (0, '') is an end of string marker, but it's getting still
slower, and quite a lot bigger, and getting so reductionistic isn't a good
thing when one class wraps another wraps another - when their comparison
methods are interrelated.
--
http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On 29/03/2011 21:32, Dan Stromberg wrote:
On Tue, Mar 29, 2011 at 12:48 PM, Ian Kelly ian.g.ke...@gmail.com
mailto:ian.g.ke...@gmail.com wrote:
On Tue, Mar 29, 2011 at 1:08 PM, Chris Angelico ros...@gmail.com
mailto:ros...@gmail.com wrote:
On Wed, Mar 30, 2011 at 5:57 AM, MRAB pyt...@mrabarnett.plus.com
mailto:pyt...@mrabarnett.plus.com wrote:
You would have to do more than that.
For example, A, but if you negate both strings you get
\xBE, not \xBE.
Strings effectively have an implicit character at the end that's less
than any other character. Easy fix: Append a character that's greater
than any other. So A becomes \xFF \xBE\xFF.
Still not going to be particularly efficient.
Not to mention that it still has bugs:
\0
\xff \xff\xff
--
http://mail.python.org/mailman/listinfo/python-list
It probably could be a list of tuples:
'abc' - [ (1, chr(255 - ord('a')), (1, chr(255 - ord('b')), (1, chr(255
- ord('c')), (0, '') ]
...where the (0, '') is an end of string marker, but it's getting
still slower, and quite a lot bigger, and getting so reductionistic
isn't a good thing when one class wraps another wraps another - when
their comparison methods are interrelated.
I think I've found a solution:
class NegStr:
def __init__(self, value):
self._value = value
def __lt__(self, other):
return self._value other._value
so when sorting a list of tuples:
def make_key(t):
return NegStr(t[0]), t[1]
items = [(one, 1), (two, 2), (three, 3), (four, 4),
(five, 5)]
items.sort(key=make_key)
--
http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Tue, Mar 29, 2011 at 5:06 PM, MRAB pyt...@mrabarnett.plus.com wrote: I think I've found a solution: class NegStr: def __init__(self, value): self._value = value def __lt__(self, other): return self._value other._value IOW: cmp_to_key(lambda x, y: -cmp(x, y)) This is still just taking a cmp function and dressing it up as a key. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Ian Kelly ian.g.ke...@gmail.com writes: cmp_to_key(lambda x, y: -cmp(x, y)) cmp_to_key(lambda x, y: cmp(y, x)) -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Fri, Mar 25, 2011 at 10:40:03PM +, Steven D'Aprano wrote: On Fri, 25 Mar 2011 13:56:23 +0100, Antoon Pardon wrote: Look we are provided with the cmp_to_key function. If something doesn't work with that function or performs realy bad with that function, it means either the function has a bug in it or something the function relies on, has a bug in it. In either case you just fix the bug. No, it means that the trade-offs made by cmp_to_key, or by sorting with a key function, do not interact well with what you are trying to do. So and if you find something like that, what stops the dev team from saying that python is just not suited to deal with such a problem? Specifically as your analogy points in that direction as your dillema has a strategy that is good for python on the one hand and a problem not suitable for python on the other hand. [analgy of python lists, that double in size when full vs memory management on a 8K device ] So the question is, being completely general, as there any *real-world* (not artificial, made-up) uses where sorting with a comparison function works but a key function doesn't? I'm not limiting you to cases where you have a shortage of memory but lots of time available. There may be other design choices where key-based sorting sucks. We already know that some people just prefer the look and feel of writing and using cmp functions. Is there anything else? Asking for *real-world* uses is just how the python community smothers requests. Should someone come with a use case, I expect the following to happenr: One will go over the case with magnifying glasses in search for any possible way in which it could be rewritten so as not to need the cmp anyway. This will probably not be a big problem because the person with the use case will probably have given a simplified example to explain the overal picture. However most people responding will ingnore the overal picture and focus on details that may very well work on the simplified example but may not in the specific real-world case. But since going over all details to argue the case will be too time consuming, in the end the person with the use case will simply fail to convince. Forcing people to use a key-function, will produce cases where python will ask for more memory and take longer to sort, than allowing to provide a cmp function. This for the simple reason that for some order-functions, the only way to write a key-function will be to write a specific class, that will implement the order with the same kind of code that otherwise would have been put in the cmp function, making the trade off: memory use vs speed no longer a trade off. If the existance alone of such cases isn't enough to reverse the decision about the cmp-argument of the sort-method. My guess is that nothing will. -- Antoon Pardon -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Mon, 28 Mar 2011 11:06:20 +0200, Antoon Pardon wrote: On Fri, Mar 25, 2011 at 10:40:03PM +, Steven D'Aprano wrote: On Fri, 25 Mar 2011 13:56:23 +0100, Antoon Pardon wrote: Look we are provided with the cmp_to_key function. If something doesn't work with that function or performs realy bad with that function, it means either the function has a bug in it or something the function relies on, has a bug in it. In either case you just fix the bug. No, it means that the trade-offs made by cmp_to_key, or by sorting with a key function, do not interact well with what you are trying to do. So and if you find something like that, what stops the dev team from saying that python is just not suited to deal with such a problem? There are lots of problems that Python is not well-suited for. It is not well-suited for writing the driver to a graphics card. It is not well-suited to calculating pi to a trillion decimal places. It is not well-suited for application development on a device with 512K of RAM. It is not well-suited for copying C idioms exactly, or Java, or Perl. It is not well-suited for doing a lot of bit-twiddling very quickly. That's why there is more than one programming language: because no one language can be well-suited for everything. [...] Asking for *real-world* uses is just how the python community smothers requests. 99% of requests should be smothered. That is a *good thing*. Insisting on real-world uses prevents the developers from wasting their valuable time bloating Python with thousands of badly desired anti-features that are slow, unreliable, buggy, useless or unpopular. [...] Forcing people to use a key-function, will produce cases where python will ask for more memory and take longer to sort, than allowing to provide a cmp function. More memory yes; take longer to sort, almost certainly not (except, perhaps, for a very narrow window where the addition of a key causes the application to page out to virtual memory, but a comparison function wouldn't -- and even then, the key function might still be faster). But we've already covered the more memory angle to death. We already understand that. This is why I have asked if there are any *other* use- cases for a comparison function. Dan Stromberg has suggested that another use-case would be lazy- comparisons, where you might not be able to generate a single key cheaply, but you can perform a comparison lazily. This for the simple reason that for some order-functions, the only way to write a key-function will be to write a specific class, that will implement the order with the same kind of code that otherwise would have been put in the cmp function, There is a recipe for that on ActiveState's website, google for cmp_to_key. That recipe has been added to Python's standard library for version 3.2. Please stop flogging that dead horse. making the trade off: memory use vs speed no longer a trade off. You want to use a comparison function: you can still do this. Instead of calling data.sort(cmp=myfunc) you call data.sort(key=functools.cmp_to_key(myfunc)) which is a little longer to type, and uses a little more memory, but otherwise is no worse than using cmp. If the existance alone of such cases isn't enough to reverse the decision about the cmp-argument of the sort-method. My guess is that nothing will. Existence of which cases? Made-up imaginary cases that don't actually happen? Silly cases that can easily be converted to key comparisons, and run much faster if you do so? Trivial cases that can trivially be converted to key functions? I can't be bothered to write a key function, to paraphrase one of the posters in this thread -- these are hardly convincing. So far I have one actual use-case: Guido's example of sorting a huge list where the amount of time it takes doesn't matter; plus one hypothetical but plausible use-case: sorting data where the comparison is expensive but lazy. (And to give Carl Banks the benefit of the doubt, one that I don't understand, about interfacing with an obscure C library.) -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Fri, Mar 25, 2011 at 10:06:59PM +, Steven D'Aprano wrote: On Fri, 25 Mar 2011 10:21:35 +0100, Antoon Pardon wrote: On Thu, Mar 24, 2011 at 11:49:53PM +, Steven D'Aprano wrote: On Thu, 24 Mar 2011 17:47:05 +0100, Antoon Pardon wrote: However since that seems to be a problem for you I will be more detailed. The original poster didn't ask for cases in which cmp was necessary, he asked for cases in which not using cmp was cumbersome. I'm the original poster, and that's not what I said. I said: If anyone has any use-cases for sorting with a comparison function that either can't be written using a key function, or that perform really badly when done so, this would be a good time to speak up. You'll notice that I said nothing about whether writing the code was easy or cumbersome, and nothing about readability. Well fine. I should have realised the question was just a pretense and that there really never was any intention to consider the reactions, because the answer is already fixed. Of course a key function can always be written, it may just need a specific class to implement the specific order. Likewise there is no reason to expect the order-functions to preform worse when implemented in a class, rather than in a function. The reason Guido is considering re-introducing cmp is that somebody at Google approached him with a use-case where a key-based sort did not work. The use-case was that the user had masses of data, too much data for the added overhead of Decorate-Sort-Undecorate (which is what key does), but didn't care if it took a day or two to sort. The question is: Why do you need a use case for this? Should the decision to remove the cmp-argument, rest only on the fact whether or not current users are processing such masses of data without considering the possibility of future users processing such masses of data? Isn't it obvious that if you force the users into using more memory, that sooner or later, there will be a user for which this demand of extra memory will be a problem? The question is also, whether this really is a use case. Why don't you advise this person to buy a computer with more memory? One could simply see this as a problem of too little resources instead of a problem for python. IMO, the dev-team can look at this in two different ways. (1) If the machine has enough resource to contain enormous lists of data, then it should be possible to sort these data with only little extra memory. or (2) Sorting will take extra memory proportional to the length of the list you will try to sort. If your computer doesn't have this extra memory, your computer just lacks the resource to sort this large data with python. It is this design decision that should guide whether or not to have a cmp-argument. Not the fact whether or not there are now users with a data set that still fits into a python program on their computer but for which this data set is too large for python to sort on that computer. So there is at least one use-case for preferring slowly sorting with a comparison function over key-based sorting. I asked if there any others. It seems not. How about the possibility of cases where it isn't a choice between slowly sorting with a comparison function over a faster key-based sorting but where the choice is over sorting with a comparison function over a slower and more memory consuming key-based sorting? I tried to sort lists of 1 elemens. Each element a tuple two items that was to be sorted first according to the first item in ascending order, then according to the second item in descending order. This was done on python 2.6 so I had to write my own cmp_to_key function. I then sorted these lists of 1 elements a 1000 times with the following methods. 1) a cmp-function 2) key = cmp_to_key 3) key = wrapper class around cmp-function 4) key = class specifically written to implement the ordering. This was done, once with number tuples and and once with string tuples. The result in seconds numbers strings 1 63.8 74.3 2 155.3 169.6 3 156.1 170.2 4 96.2 104.4 Now you can of course ignore those numbers because they don't come from a real world example and thus consider such numbers unimportant until someone in the future has a case similar enough with the above and wonders why sorting his data is so slow. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Antoon Pardon antoon.par...@rece.vub.ac.be wrote: I tried to sort lists of 1 elemens. Each element a tuple two items that was to be sorted first according to the first item in ascending order, then according to the second item in descending order. This was done on python 2.6 so I had to write my own cmp_to_key function. I then sorted these lists of 1 elements a 1000 times with the following methods. 1) a cmp-function 2) key = cmp_to_key 3) key = wrapper class around cmp-function 4) key = class specifically written to implement the ordering. This was done, once with number tuples and and once with string tuples. The result in seconds numbers strings 1 63.8 74.3 2 155.3 169.6 3 156.1 170.2 4 96.2 104.4 Now you can of course ignore those numbers because they don't come from a real world example and thus consider such numbers unimportant until someone in the future has a case similar enough with the above and wonders why sorting his data is so slow. Out of interest, what happens to your timings if you do the sort just using simple key functions? Since you've got two keys you will of course need to make two calls to 'sort'. e.g. data.sort(key=itemgetter(1), reverse=True) data.sort(key=itemgetter(0)) -- Duncan Booth http://kupuguy.blogspot.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Mon, 28 Mar 2011 14:39:04 +0200, Antoon Pardon wrote: I tried to sort lists of 1 elemens. Each element a tuple two items that was to be sorted first according to the first item in ascending order, then according to the second item in descending order. This was done on python 2.6 so I had to write my own cmp_to_key function. I then sorted these lists of 1 elements a 1000 times with the following methods. Thank you for spending the time to get some hard data, but I can't replicate your results since you haven't shown your code. Rather than attempt to guess what you did and duplicate it, I instead came up with my own timing measurements. Results are shown here, my code follows below: [steve@sylar ~]$ python2.7 sort_test.py Comparison function: 12.1256039143 Key function: 3.51603388786 Double sort: 2.33165812492 cmp_to_key: 28.1129128933 By far the best result comes from two calls to sort. Not far off is the key function solution. (Admittedly, coming up with a key function for non- numeric data would be challenging.) The comparison function, and the cmp_to_key solution, are clearly *much* slower. I may do another test with larger lists, and leave it running overnight and see what happens. [code] from random import random, shuffle from operator import itemgetter from timeit import Timer from functools import cmp_to_key def make_data(n): x = range(n)*2 y = range(n)*2 shuffle(x) shuffle(y) data = zip(x, y) assert len(data) == 2*n return data # Sort according to the first item in ascending order, then by second item # in descending order: def cmp_func(t1, t2): return cmp(t1[0], t2[0]) or cmp(t2[1], t1[1]) def key_func(t): return (t[0], -t[1]) def sorter(d): d = sorted(d, key=itemgetter(1), reverse=True) return sorted(d, key=itemgetter(0)) # Check that all sort methods give the same result. data = make_data(20) assert sorted(data, cmp=cmp_func) == sorted(data, key=key_func) == \ sorter(data) == sorted(data, key=cmp_to_key(cmp_func)) data = make_data(5000) assert len(data) == 1 setup = from __main__ import cmp_to_key, cmp_func, key_func, sorter, data t1 = Timer(sorted(data, cmp=cmp_func), setup) t2 = Timer(sorted(data, key=key_func), setup) t3 = Timer(sorter(data), setup) t4 = Timer(sorted(data, key=cmp_to_key(cmp_func)), setup) print Comparison function:, print min(t1.repeat(number=100, repeat=7)) print Key function:, print min(t2.repeat(number=100, repeat=7)) print Double sort:, print min(t3.repeat(number=100, repeat=7)) print cmp_to_key:, print min(t4.repeat(number=100, repeat=7)) [end code] -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Mon, Mar 28, 2011 at 11:06:20AM +0200, Antoon Pardon wrote: Asking for *real-world* uses is just how the python community smothers requests. It's quite a common strategy, I've seen it used in many contexts. Which doesn't make it any more acceptable of course. Should someone come with a use case, I expect the following to happenr: One will go over the case with magnifying glasses in search for any possible way in which it could be rewritten so as not to need the cmp anyway. This will probably not be a big problem because the person with the use case will probably have given a simplified example to explain the overal picture. However most people responding will ingnore the overal picture and focus on details that may very well work on the simplified example but may not in the specific real-world case. But since going over all details to argue the case will be too time consuming, in the end the person with the use case will simply fail to convince. An accurate description of the process. Forcing people to use a key-function, will produce cases where python will ask for more memory and take longer to sort, than allowing to provide a cmp function. This for the simple reason that for some order-functions, the only way to write a key-function will be to write a specific class, that will implement the order with the same kind of code that otherwise would have been put in the cmp function, making the trade off: memory use vs speed no longer a trade off. If cmp_to_key() would do some (probably impossible) magic - factoring the user's cmp() into a non-trivial key mapping and a simpler and more efficient compare - then it would make sense. But it doesn't do such a thing. The 'key' it uses is just a copy, and the compare that will be used finally is just the user's one. Nothing gained, just overhead added. If the existance alone of such cases isn't enough to reverse the decision about the cmp-argument of the sort-method. My guess is that nothing will. Sadly enough, that is probably a good guess. Ciao, -- FA -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Mar 28, 8:43 am, Steven D'Aprano steve +comp.lang.pyt...@pearwood.info wrote: Thank you for spending the time to get some hard data, but I can't replicate your results since you haven't shown your code. Rather than attempt to guess what you did and duplicate it, I instead came up with my own timing measurements. Results are shown here, my code follows below: [steve@sylar ~]$ python2.7 sort_test.py Comparison function: 12.1256039143 Key function: 3.51603388786 Double sort: 2.33165812492 cmp_to_key: 28.1129128933 By far the best result comes from two calls to sort. Not far off is the key function solution. (Admittedly, coming up with a key function for non- numeric data would be challenging.) The comparison function, and the cmp_to_key solution, are clearly *much* slower. On Mar 28, 8:43 am, Steven D'Aprano steve +comp.lang.pyt...@pearwood.info wrote: Thank you for spending the time to get some hard data, but I can't replicate your results since you haven't shown your code. Rather than attempt to guess what you did and duplicate it, I instead came up with my own timing measurements. Results are shown here, my code follows below: [steve@sylar ~]$ python2.7 sort_test.py Comparison function: 12.1256039143 Key function: 3.51603388786 Double sort: 2.33165812492 cmp_to_key: 28.1129128933 By far the best result comes from two calls to sort. Not far off is the key function solution. (Admittedly, coming up with a key function for non- numeric data would be challenging.) The comparison function, and the cmp_to_key solution, are clearly *much* slower. There is less here than meets the eye. The cmp-function dispatch in cmp_to_key() is written in pure Python. Essentially, the timings are showing the already well-known fact that call forwarding is faster in C than in pure python. Each of the O(n log n) comparisons is run through pure Python code that like this: def __lt__(self, other): # mycmp is nested scope variable pointing to # the user-defined cmp-function return mycmp(self.obj, other.obj) 0 I intend to add a C version of cmp_to_key() so that no trips around the eval-loop are needed. It should be about as efficient as bound_method dispatch (very fast), leaving the user-supplied cmp- function as the dominant cost in the handful of cases where the superfast O(n) key-function approach can't be used for some reason. The memory overhead should either stay the same or drop slightly (currently at 14 bytes per element on a 32-bit build and 28 bytes per element on a 64-bit build). Also note that running timings on Py2.7 instead of Py3.2 disadvantages key-functions. In 2.7, key-functions use sortwrapper objects which were removed in 3.2 (saving memory and reducing dispatch time considerably). Also, in Py2.7 cmp logic is called even when a key- function is defined (because key-function logic was wrapped around the already existing sort with cmp-logic) so you pay the price of both (i.e. creating a 2-tuple for cmp arguments on every call). In Py3.x the cmp logic is gone, making the remaining key-function logic even faster. IOW, key-function logic is way faster and more space efficient in 3.2. One of the implications of the last paragraph is that if 2.7 style cmp- logic were reinstated in 3.3, not only would it clutter the API with two-ways-to-do-it, it would also disadvantage make the common case (using key-functions) run much more slowly. In other words, the performance mavens will have shot themselves (and us) in the foot. Raymond -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Steven D'Aprano steve+comp.lang.pyt...@pearwood.info writes: There are lots of problems that Python is not well-suited for. It is not well-suited for writing the driver to a graphics card. It is not well-suited to calculating pi to a trillion decimal places. ... But sorting moderate amounts of data on complicated criteria is an area Python IS supposed to be well-suited for. 99% of requests should be smothered. That is a *good thing*. We're not talking about a request for a feature to be added. There was an existing feature that worked perfectly well, that was removed for some ideological reason that makes no sense to me, with some slight allusions to technical reasons that I've also never completely understood. More memory yes; take longer to sort, almost certainly not (except, perhaps, for a very narrow window where the addition of a key causes I don't see how using something like cmp_to_key could ever fail to slow the program down. It's calling the exact same comparison function the same number of times, but also creating and releasing a pile of class instances, and doing a lot of runtime lookups to access the comparison function in each instance on each comparison. data.sort(key=functools.cmp_to_key(myfunc)) which is a little longer to type, and uses a little more memory, but otherwise is no worse than using cmp. I don't believe that for one second. Have you actually timed it? Like, sort an array of 10 random (int,int) pairs ascending on the first int and descending on the second, using a simple comparison function and using cmp_to_key? That's Anton's example, and by coincidence that exact situation came up in something I'm hacking on RIGHT NOW. I have (roughly) a list of transactions in the form (person's name, timestamp, transaction info) and I want to print a report sorted alphabetically by person, listing each person's transactions most-recent-first. Of course I was able to concoct a solution with a few moments of head-scratching, but using a cmp argument is much more natural and transparent in my opinion. DSU is a clever and useful design pattern, but comparison sorting is what all the sorting textbooks are written about. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Mon, Mar 28, 2011 at 6:58 PM, Paul Rubin no.email@nospam.invalid wrote: DSU is a clever and useful design pattern, but comparison sorting is what all the sorting textbooks are written about. Actually, even though I wrote one program that could almost benefit from cmp sorting that might have trouble with key sorting, I think it was probably the right decision to remove cmp sorting. I say almost, because I've since rewritten it with a specialized sort that gives it a much better asymptotic runtime and better fault tolerance, so what Python 3 does out of the box for sorting won't matter to it much. I say it's probably the right decision, because it's almost always going to be faster to use a key sort, at least in CPython = 3.2, and it's almost never less functional to use a key sort. Using a key sort also opens the door to a radixsort, not that radixsort is really that interesting as sorts go (It's still pretty much O(nlogn), statements to the contrary notwithstanding), but it's nice to know that the possibility is there. If you run across one of the infrequent, specialized sorting needs that require a cmp sort, there's not really anything wrong with using cmp_to_key or including your own sort routine. I put a bunch of Python sorts at http://stromberg.dnsalias.org/cgi-bin/viewvc.cgi/sorts/compare/trunk/?root=svn- but I've not needed to add cmp -or- key comparisons to them yet - so far they just use , , ==, etc. The program I mentioned above doesn't use any of these - it has its own, relatively unique sort algorithm - mostly to collapse same-valued elements into a list to avoid re-compares, and to eliminate elements that give errors during the sort. I leave off with this: Python -must- resist the temptation to become all things to all people, because no language can ever truly -be- all things to all people. I much prefer having a nice small core language to having a kitchen sink language. -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
Steven D'Aprano wrote: On Fri, 25 Mar 2011 10:21:35 +0100, Antoon Pardon wrote: The reason Guido is considering re-introducing cmp is that somebody at Google approached him with a use-case where a key-based sort did not work. The use-case was that the user had masses of data, too much data for the added overhead of Decorate-Sort-Undecorate (which is what key does), but didn't care if it took a day or two to sort. So there is at least one use-case for preferring slowly sorting with a comparison function over key-based sorting. I asked if there any others. It seems not. That is one use case for preferring cmp. Another use case is I can easily think of how to write the cmp function and not the key function and I don't care about either speed or memory usage. There's more to practicality than code execution speed. -- --OKB (not okblacke) Brendan Barnwell Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail. --author unknown -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Fri, 25 Mar 2011 16:04:02 -0700, Carl Banks wrote: On Mar 25, 3:06 pm, Steven D'Aprano steve +comp.lang.pyt...@pearwood.info wrote: The reason Guido is considering re-introducing cmp is that somebody at Google approached him with a use-case where a key-based sort did not work. The use-case was that the user had masses of data, too much data for the added overhead of Decorate-Sort-Undecorate (which is what key does), but didn't care if it took a day or two to sort. So there is at least one use-case for preferring slowly sorting with a comparison function over key-based sorting. I asked if there any others. It seems not. 1. You asked for a specific kind of use case. Antoon gave you a use case, you told him that wasn't the kind of use case you were asking for, then you turn around and say I guess there are no use cases (without the mentioning qualification). That is astonishing. Did you read my post before going on the attack? I know that there is a grand tradition of adversarial argument on Usenet, but I am not your enemy here. First you wrongly accuse me of stating that there are no use cases, without qualification, when the very text you quote has me giving a use- case. Nor did I say there are no use cases -- I said that I didn't think there were any *others*. You even quote me saying this! Then you go on to wrongly claim: 3. You evidently also overlooked the use-case example posted on Python- dev that you followed up to. No, I did not overlook it. I explicitly described it twice. Once in the very text you quoted above, and once early in the thread when I wrote: Guido himself has mentioned one such good use for a comparison function when sorting. Use of a key function trades off memory for time, while sorting with a comparison function makes the opposite trade off, using more time for the sake of saving memory. 2. I posted two use cases in this thread that fit your criteria, and you followed up to that subthread so you most likely read them. Here they are again so you won't overlook them this time: You have are given an obscure string collating function implented in a C library you don't have the source to. (Fits your criterion can't be done with key=.) I must admit I don't understand this use-case. What is a string collating function, and what does this have to do with sorting? I could try to guess from the plain English meaning of the words, but if I get it wrong, that wouldn't be fair to you. Perhaps an example would help. I'm sitting at an interactive session and I have a convenient cmp function but no convenient key, and I care more about the four minutes it'd take to whip up a clever key function or an adapter class than the 0.2 seconds I'd save to on sorting time. (Fits your criterion performs really badly when done so.) I can't be bothered to write a key function is not equivalent to I have a key function, but it is too slow to use. In any case, from Python 3.2 onwards, a key function is never more than one import and one function call away. (In 3.1, you're stuck writing your own adapter, but in 3.1 sort doesn't take a cmp function anyway, so either way you're stuck. The earliest it could be reintroduced, if at all, is in 3.3.) Call me crazy, but you seem to be overlooking a lot of things in your zeal to prove your point. And what point would that be? Carl, anybody could have asked the question I have asked. Anyone is free to summarize the best of the arguments -- or all of them -- and post them to python-dev. If you think I have some ulterior motive in doing this, please feel free to champion the cmp argument to python-dev instead. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Guido rethinking removal of cmp from sort method
On Mar 25, 2:00 pm, Stefan Behnel stefan...@behnel.de wrote: Westley Martínez, 25.03.2011 14:39: On Fri, 2011-03-25 at 07:11 +0100, Stefan Behnel wrote: Steven D'Aprano, 25.03.2011 06:46: On Thu, 24 Mar 2011 18:32:11 -0700, Carl Banks wrote: It's probably the least justified builtin other than pow. I don't know about that. Correctly, efficiently and *quickly* implementing the three-argument version of pow is exactly the sort of thing that should be in the built-ins, or at least the standard library. I think that touches it already. We have a math module, so why is there a *builtin* for doing special math? How much code is there really that only uses pow() and does not import math? pow() and math.pow() are different. I don't find that a good excuse for pow() being a builtin. Why not just rename it to math.powmod instead? +1 (modulo bikeshedding about the name). I've long thought that three- argument pow belongs in the standard library rather than the core. And then the existence of the ** operator obviates the need for two- argument pow. Worse than pow itself is the fact that the __pow__ special method takes an optional 3rd argument, which apart from unnecessarily complicating Python's internal, also leads to horrors like Decimal's (IMO inappropriate) support for 3-argument pow. -- Mark -- http://mail.python.org/mailman/listinfo/python-list
