Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
Ondrej Certik wrote: Are we able to provide an actual patch to Python that implements this? If so, then I am. Imho the proposal should come with an actual patch, otherwise it's difficult to judge it. Your better off with writing a PEP first. In order to implement the proposal you've to make changes to the parser and tokenizer - perhaps to the grammar, too. The code is rather complex and tricky - and not very well documented. Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
Hi, with Andrew's permission, I am starting a new thread, where our discussion is ontopic. :) My original question was, that I would like to override 1+1 to return MyClass(1, 1) or something. Robert said it would break other libraries and Andrew said this: On Mon, Aug 18, 2008 at 9:23 PM, Andrew Dalke [EMAIL PROTECTED] wrote: There are basically two possible options: 1) global overloading of + 2) allow to tell Python that 1 is not its int, but ours Integer. BTW, Sage just preparses Python for exactly this reason and substitutes all numbers like 1 with Integer(1). This really sucks imho. How would you like to do it? Any solution I can think of would cause huge problems. For example, being able to change int.__add__ would cause systemic problems throughout the code base and would prevent, or very much hinder, migration of Python code to C. Agree. Changes to support a different constructor for basic types, on a per-module basis has its own problems. When are the new types specified? In the module itself or by the code which imports the module? Can the definitions be changed on a scope-by-scope level? Now, I can imagine a Python with builtin multimethod dispatch defined via static scoping, so that def __add__(left:int, right:int): return __builtin__.__add__(__builtin__.__add__(left, right), 2) def __int__(s): return Decimal(s) might work, but even here there's a problem because the 2 in __add__ gets replaced by a Decimal, when I really want it to be an integer. Not getting why is 2 replaced by Decimal if you don't want to? If you don't want it, you can just reimplement __int__(s). This would of course be only active in the module where it was defined. So I don't see how you can do it in the context of Python. In the context of a very different programming language, sure, it's possible. But that's what Sage is doing, they just preparse the input. But in SymPy we'd like to use it as a regular Python library in users programs, so imho preparsing, or custom compiling using your package is not really an option, is it? Use a different extension, like .sympy and an import hook which does the .sympy - .pyc conversion. But it's not one I recommend for real code, at least not without a lot of personal experience to see if it's worthwhile in the face of the downsides. Yes, this is a mess, this is just like preparsing. Well, not like -- this is preparsing. Ok, in the current state, you don't know either what's going to happen. If you write In [1]: x/2*3/4 you have no idea what the result is going to be, you need to analyze x.__div__() and start from there. But if you write In [2]: 1/2*3/4 currently you know it will be 0. But imho you could as well analyze the global __mul__ (or global __int__, depending on how this would be technically implemented) to see what's going to happen. I mean what is the difference between [1] and [2]? Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
On Aug 18, 2008, at 10:01 PM, Ondrej Certik wrote: with Andrew permission, I am starting a new thread, where our discussion is ontopic. :) Though I want to point out that without specific proposals of how the implementation might look, this thread will not go anywhere as it will be too distant from usable code. I sent examples to show how such a system might look, as the basis for getting a feel if it was practical. I do not think my examples are practical, but they were meant as an example of how such a proposal might look. Since I know that the Python implementation will not change to support per-module or per-scope redefinitions for 1+2 and builtin object constructors, the only feasible mechanism is through some sort of alternate grammar that compiles to either Python or directly to the Python virtual machine. One such was is through import hooks. Yes, this is a mess, this is just like preparsing. Well, not like -- this is preparsing. It's not preparsing. It's parsing. There's no pre about it. It's not a macro language. My ply4python tutorial compiles various Python-like languages to the Python virtual machine bytecode. I mean what is the difference between [1] and [2]? I want to see how you would extend Python to support such a mechanism before I worried about how to interpret it. Or in other words, the difference between [1] and [2] is that [2] can be fully evaluated through simple static analysis, while [1] cannot. BTW, this is unexpected. Python does constant folding of that expression, but only with specific settings. import dis def f(): ... print 1/2*3/4 ... dis.dis(f) 2 0 LOAD_CONST 1 (1) 3 LOAD_CONST 2 (2) 6 BINARY_DIVIDE 7 LOAD_CONST 3 (3) 10 BINARY_MULTIPLY 11 LOAD_CONST 4 (4) 14 BINARY_DIVIDE 15 PRINT_ITEM 16 PRINT_NEWLINE 17 LOAD_CONST 0 (None) 20 RETURN_VALUE from __future__ import division def f(): ... print 1/2*3/4 ... dis.dis(f) 2 0 LOAD_CONST 7 (0.375) 3 PRINT_ITEM 4 PRINT_NEWLINE 5 LOAD_CONST 0 (None) 8 RETURN_VALUE The only way I can see to do what you want requires multimethods, which don't currently exist in Python except as third-party extensions. The one I know about, from Philip J. Eby, works on a global-level, not module level, because of how registration happens, so it does not support what you would like. Andrew [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
On Mon, Aug 18, 2008 at 10:45 PM, Andrew Dalke [EMAIL PROTECTED] wrote: On Aug 18, 2008, at 10:01 PM, Ondrej Certik wrote: with Andrew permission, I am starting a new thread, where our discussion is ontopic. :) Though I want to point out that without specific proposals of how the implementation might look, this thread will not go anywhere as it will be too distant from usable code. I sent examples to show how such a system might look, as the basis for getting a feel if it was practical. I do not think my examples are practical, but they were meant as an example of how such a proposal might look. Since I know that the Python implementation will not change to support per-module or per-scope redefinitions for 1+2 and builtin object constructors, the only feasible mechanism is through some sort of alternate grammar that compiles to either Python or directly to the Python virtual machine. One such was is through import hooks. Yes, this is a mess, this is just like preparsing. Well, not like -- this is preparsing. It's not preparsing. It's parsing. There's no pre about it. It's not a macro language. My ply4python tutorial compiles various Python-like languages to the Python virtual machine bytecode. I mean what is the difference between [1] and [2]? I want to see how you would extend Python to support such a mechanism before I worried about how to interpret it. Or in other words, the difference between [1] and [2] is that [2] can be fully evaluated through simple static analysis, while [1] cannot. BTW, this is unexpected. Python does constant folding of that expression, but only with specific settings. import dis def f(): ... print 1/2*3/4 ... dis.dis(f) 2 0 LOAD_CONST 1 (1) 3 LOAD_CONST 2 (2) 6 BINARY_DIVIDE 7 LOAD_CONST 3 (3) 10 BINARY_MULTIPLY 11 LOAD_CONST 4 (4) 14 BINARY_DIVIDE 15 PRINT_ITEM 16 PRINT_NEWLINE 17 LOAD_CONST 0 (None) 20 RETURN_VALUE from __future__ import division def f(): ... print 1/2*3/4 ... dis.dis(f) 2 0 LOAD_CONST 7 (0.375) 3 PRINT_ITEM 4 PRINT_NEWLINE 5 LOAD_CONST 0 (None) 8 RETURN_VALUE The only way I can see to do what you want requires multimethods, which don't currently exist in Python except as third-party extensions. The one I know about, from Philip J. Eby, works on a global-level, not module level, because of how registration happens, so it does not support what you would like. One way to fix that would be to use the trick that Travis Oliphant told me at EuroSciPy -- hack the while (or if) statement and do the preparsing in there. So clearly the language seems to support that, so imho it could be made more official. E.g. you would write: while sympy: e = 1/2 and e would be Integer(1)/Integer(2). But anyway, it's kind of hackish and I don't know what to say more about it, besides what I already said. The problem is that I don't have time to dig more into Python internals and without it it seems I cannot provide more constructive answer besides I want some way to avoid Python reducing 1/2 to 0. Generally I believe that if there is will, it can always be done. :) Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
Ondrej Certik wrote: Ok, in the current state, you don't know either what's going to happen. If you write In [1]: x/2*3/4 you have no idea what the result is going to be, you need to analyze x.__div__() and start from there. But if you write In [2]: 1/2*3/4 currently you know it will be 0. But imho you could as well analyze the global __mul__ (or global __int__, depending on how this would be technically implemented) to see what's going to happen. I mean what is the difference between [1] and [2]? Andrew has already pointed it out very well. I like to comment on your proposal from the perspective of a Python core developer as well as the perspective of somebody who has worked with Guido for more than a year. I'd bet my life that Guido is never every going to allow it. The core types are fundemental to the Python interpreter. Even the possibility of pluggable type methods would make the implementation slower, more fragile and much more complicated. We'd have to remove several speed tricks and special cases for e.g. ints and replace them with slower implementations. But don't give up hope yet! During the alpha phase of Python 3.0 and the revamping of the decimal module, some core developers had an even better idea. We were discussing the possibility of making decimals the default for float literals. The idea was rejected eventually but it gave birth to yet another idea. What about making the *result* of a literal pluggable? The Python creates a float for the literal 1.0. Some header in a module could replace the default target 'float' with e.g. decimal.Decimal. Example syntax (rough idea): type(1.0) type 'float' with float as from decimal import Decimal type(1.0) class 'decimal.Decimal' Wouldn't that solve your general problem more elegant without breaking other modules? Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
Hi Christian, On Mon, Aug 18, 2008 at 11:22 PM, Christian Heimes [EMAIL PROTECTED] wrote: Ondrej Certik wrote: Ok, in the current state, you don't know either what's going to happen. If you write In [1]: x/2*3/4 you have no idea what the result is going to be, you need to analyze x.__div__() and start from there. But if you write In [2]: 1/2*3/4 currently you know it will be 0. But imho you could as well analyze the global __mul__ (or global __int__, depending on how this would be technically implemented) to see what's going to happen. I mean what is the difference between [1] and [2]? Andrew has already pointed it out very well. I like to comment on your proposal from the perspective of a Python core developer as well as the perspective of somebody who has worked with Guido for more than a year. I'd bet my life that Guido is never every going to allow it. The core types are fundemental to the Python interpreter. Even the possibility of pluggable type methods would make the implementation slower, more fragile and much more complicated. We'd have to remove several speed tricks and special cases for e.g. ints and replace them with slower implementations. But don't give up hope yet! During the alpha phase of Python 3.0 and the revamping of the decimal module, some core developers had an even better idea. We were discussing the possibility of making decimals the default for float literals. The idea was rejected eventually but it gave birth to yet another idea. What about making the *result* of a literal pluggable? The Python creates a float for the literal 1.0. Some header in a module could replace the default target 'float' with e.g. decimal.Decimal. Example syntax (rough idea): type(1.0) type 'float' with float as from decimal import Decimal type(1.0) class 'decimal.Decimal' Wouldn't that solve your general problem more elegant without breaking other modules? It absolutely would. Thanks very much for the email. How is your proposal (redefine literals) different to just saying to Python -- hey, just call my class when someone writes 1, e.g. proposition 2) from my first email? Or am I missing something. I agree with the technical reasonings, why some particular solution is not good. I.e. I didn't do any proposal, I am just trying to find a way, so that we don't have to always type In [3]: Integer(1)/2 * x sometimes, but In [4]: x/2 some other times. If you know what I mean. Both do the same thing, but [1] is very annoying to write and a source of common mistakes that people do with SymPy, it simply returns 0. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
On Aug 18, 2008, at 11:22 PM, Christian Heimes wrote: Example syntax (rough idea): type(1.0) type 'float' with float as from decimal import Decimal type(1.0) class 'decimal.Decimal' When would this with float ... considered valid? For example, could I define things before asking for a redefinition? def f(a=1.0): print a+2.0 with float as from decimal import Decimal In this case would the 1.0 be a float and the 2.0 be a decimal? Or would they both be floats? Would the following be allowed? def f(x): with float as from decimal import Decimal return 1.0 and would that affect things in the function scope or the entire module scope? What if I stuck in a global int in that function? What about the following, which uses gmpy if it's available, otherwise uses decimal. try: import gmpy # Pretend there's a bug in some versions # of the library, and don't use gmpy # if that bug is present. if gmpy.mpf(3.0) == 3.0: with float as from gmpy import gmpy.mpf else: # work around the hypothetical bug raise ImportError except ImportError: with float as from decimal import Decimal Hmm.. though this could be handled with some support library, making the result be with float as support_library.the_float_to_use The simplest is that if that statement type appears in the module at all then int/float/complex/string/ dict/list creation goes through a module function, and the statement simply redefines that function. This would slow down the entire module, but if it's what the developer wanted... But I really don't like the unexpected slow down and I think that will be a constant gotcha, with people rewriting code from sum(1 for x in data if x 100) into something like one = 1 one_hundred = 100 # or possibly: one = __builtin__.int(1) sum(one for x in data if x one_hundred) in order to eek out performance by not calling new_int(1) so many times. Also, in the use-case for SymPy it means that all modules would start with: with float as from SymPy import float with int as from SymPy import int with complex as from SymPy import complex which makes for tedious boilerplate. Better might be with (float, int, complex) as from SymPy import (float, int, complex) If there's support for with str as then how does one create an actual Python string in a module where string is redefined? For example, with str as from MyModule import string open(__builtins__.str(/path/to/nowhere)) wouldn't necessarily work because /path/to gets converted to something else, and that something else might not be convertible back to a Python string. Blowing my own horn again, I wrote python4ply precisely to support experimentation like this. We can discuss pros and cons but with a change this extensive it's hard to judge the usefulness with without some real-world experience. And I didn't want to wait for PyPy. ;) Andrew [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
Ondrej Certik wrote: Hi Christian, On Mon, Aug 18, 2008 at 11:22 PM, Christian Heimes [EMAIL PROTECTED] wrote: Ondrej Certik wrote: Ok, in the current state, you don't know either what's going to happen. If you write In [1]: x/2*3/4 you have no idea what the result is going to be, you need to analyze x.__div__() and start from there. But if you write In [2]: 1/2*3/4 currently you know it will be 0. But imho you could as well analyze the global __mul__ (or global __int__, depending on how this would be technically implemented) to see what's going to happen. I mean what is the difference between [1] and [2]? Andrew has already pointed it out very well. I like to comment on your proposal from the perspective of a Python core developer as well as the perspective of somebody who has worked with Guido for more than a year. I'd bet my life that Guido is never every going to allow it. The core types are fundemental to the Python interpreter. Even the possibility of pluggable type methods would make the implementation slower, more fragile and much more complicated. We'd have to remove several speed tricks and special cases for e.g. ints and replace them with slower implementations. But don't give up hope yet! During the alpha phase of Python 3.0 and the revamping of the decimal module, some core developers had an even better idea. We were discussing the possibility of making decimals the default for float literals. The idea was rejected eventually but it gave birth to yet another idea. What about making the *result* of a literal pluggable? The Python creates a float for the literal 1.0. Some header in a module could replace the default target 'float' with e.g. decimal.Decimal. Example syntax (rough idea): type(1.0) type 'float' with float as from decimal import Decimal type(1.0) class 'decimal.Decimal' Wouldn't that solve your general problem more elegant without breaking other modules? It absolutely would. Thanks very much for the email. How is your proposal (redefine literals) different to just saying to Python -- hey, just call my class when someone writes 1, e.g. proposition 2) from my first email? Or am I missing something. I agree with the technical reasonings, why some particular solution is not good. I.e. I didn't do any proposal, I am just trying to find a way, so that we don't have to always type In [3]: Integer(1)/2 * x sometimes, but In [4]: x/2 some other times. If you know what I mean. Both do the same thing, but [1] is very annoying to write and a source of common mistakes that people do with SymPy, it simply returns 0. It should be mentioned here that this is exactly what the SAGE preparser does -- all 2 turns into Integer(2), and that idea definitely seems to work for them. (And so if you want Python behaviour in SAGE, you simply do Integer = int, while normally 1/2 becomes a rational object in QQ because one object in ZZ is divided by another one). Pluggable literals seems like a much better idea to me, not from pragmatic but from design reasons -- I would prefer them even in the (very unlikely) case that they were slower and harder to implement than overriding the operators for the builtin types. (But I'm just a bystander here.) Dag Sverre ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
Andrew Dalke wrote: When would this with float ... considered valid? [long posting] Oh h... what have I done ... *g* Slow down, please. For now there are no concrete plans what-so-ever to implement the feature in the near future. Some developers have expressed their interest in a way to alter the resulting type of a literal. It was my attention to show you, that we have discussed the idea, too. Now for the with type as from import syntax. I came up with the syntax idea about an hour ago. I tried to come up with some nice syntax that reuses existing keywords. IMHO it has a nice ring. Other possibilities I came up with: def float as factory def float as from module import factory with float yield factory with float yield from module import factory After some careful thinking I'm in favor of with ... yield It's less ambiguous and can't be mistaken for with open(filename) as fh. The ideas needs a good PEP. You are definitely up to something. You also came up with a list of possible issues and corner cases. Are you interested in pursuing the proposal? *wink* Christian ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
On Tue, Aug 19, 2008 at 1:06 AM, Christian Heimes [EMAIL PROTECTED] wrote: Andrew Dalke wrote: When would this with float ... considered valid? [long posting] Oh h... what have I done ... *g* Slow down, please. For now there are no concrete plans what-so-ever to implement the feature in the near future. Some developers have expressed their interest in a way to alter the resulting type of a literal. It was my attention to show you, that we have discussed the idea, too. Now for the with type as from import syntax. I came up with the syntax idea about an hour ago. I tried to come up with some nice syntax that reuses existing keywords. IMHO it has a nice ring. Other possibilities I came up with: def float as factory def float as from module import factory with float yield factory with float yield from module import factory After some careful thinking I'm in favor of with ... yield It's less ambiguous and can't be mistaken for with open(filename) as fh. The ideas needs a good PEP. You are definitely up to something. You also came up with a list of possible issues and corner cases. Are you interested in pursuing the proposal? *wink* Are we able to provide an actual patch to Python that implements this? If so, then I am. Imho the proposal should come with an actual patch, otherwise it's difficult to judge it. Ondrej ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
On Aug 19, 2008, at 1:06 AM, Christian Heimes wrote: [long posting] Oh h... what have I done ... *g* *shrug* I write long emails. I've been told that by several people. It's probably a bad thing. The ideas needs a good PEP. You are definitely up to something. You also came up with a list of possible issues and corner cases. Are you interested in pursuing the proposal? *wink* No. I'm doing this to head off endless discussion about some proposed hypothetical future Python. If someone wants to experiment with new operators, new ways of handling int/float/whatever constructions, new lexical forms, or whatever then IMNSHO the best thing is to implement it and see if it's actually useful. In most cases there are seriously hard questions that are hand-waved away, because it's more fun to talk about language design than to implement one and because it's hard to tweak CPython to try out new things. Hence I wrote python4ply and an extensive tutorial specifically so people could easily jump into the implementation details, find the sticky points, then either quickly reject a bad idea or iterate to get a good solution. Many of these changes can be done with less than a couple of days of work. Then to show that there are sticky points, I listed a few. BTW, it's *fun* to modify an existing language and afterwards it you know a secret - that programming languages are just flimsy facades held together by a shared hallucination. Like in a dream, change things too much or leave gaps and people notice the illogic, wake up, and look elsewhere for refuge from harsh digital machine reality. Hmmm, it really is too late for me. Andrew [EMAIL PROTECTED] ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
On Tue, Aug 19, 2008 at 01:57:20AM +0200, Andrew Dalke wrote: BTW, it's *fun* to modify an existing language and afterwards it you know a secret - that programming languages are just flimsy facades held together by a shared hallucination. Like in a dream, change things too much or leave gaps and people notice the illogic, wake up, and look elsewhere for refuge from harsh digital machine reality. That is the quote of the month for sure. Scott -- Scott M. RansomAddress: NRAO Phone: (434) 296-0320 520 Edgemont Rd. email: [EMAIL PROTECTED] Charlottesville, VA 22903 USA GPG Fingerprint: 06A9 9553 78BE 16DB 407B FFCA 9BFA B6FF FFD3 2989 ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] global overloading of 1+1 - MyClass(1, 1)
On Aug 18, 2008, at 6:57 PM, Andrew Dalke wrote: ... BTW, it's *fun* to modify an existing language and afterwards it you know a secret - that programming languages are just flimsy facades held together by a shared hallucination. Like in a dream, change things too much or leave gaps and people notice the illogic, wake up, and look elsewhere for refuge from harsh digital machine reality. I think you meant to say shared abstraction, right Andrew ... (call the guards!) Travis ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
