Re: Python method overloading

[Leopold Toetsch]

Sam Ruby <[EMAIL PROTECTED]> wrote:
> Leopold Toetsch wrote:

>> # find_method class 'A' method '__absolute': Sub
>> # Calling sub '__absolute'

> But only for classes that inherit from delegate.

Yes of course. Objects's derived from ParrotObject (i.e. Parrot standard
objects) dispatch to overloaded functions via the meta-class helper
class "delegate". Standard PMCs call pmc->vtable->absolute().

But infix operations like the "add" opcode dispatch via mmd_dispatch_*
and have no delegate functionality.

I've posted a graphic of all these possibilities in one of the mails.

This is the reason why I'm prefering to replace all with just one
scheme that works, i.e. doing full dynamic dispatch at the opcode level.

> People writing Python to Parrot translators need to know Parrot
> internals.  People who merely write in Python should not.

Why on earth should Python writers know Parrot internals? Why do you
come to such a conclusion?

>>>... (BTW, a combined instantiate method
>>>does not map well to Python which has separate __new__ and __init__
>>>methods.
>>
>> We have the "__init" hook too. This is separate.

> Ultimately, I believe that this will need to be revisited.

What is wrong? Does it not work?

>>>...  My recommendation
>>>is to stick to primitives, and simply provide a new_p_p).
>>
>> What is the second "_p" for?

> What I am thinking of is something like:

>inline op new(out PMC) {
>  $1 = VTABLE_instantiate(interpreter, $2);

There is no second argument on the "new" signature. But I presume it's a
class PMC. Well doing:

  cl = getclass "Foo"
  o = cl."instantiate"()

does exactly that - it constructs a new Integer with the default value by
calling pmc_new/init.

OTOH when you have

  cl = getclass "Foo"
  o = cl."instantiate"(5)

you have two possiblities: implement the "instantiate" vtable (or let a
user provided overridden method do the initialization) or do the init
stuff in "__init", which get's again the arguments according to pdd03.

>   If you then want to init the PMC you have
> obtained, you can use the separate init hook (with a variable number of
> arguments).

This is working already.

.sub main
.local pmc cl, o
cl = subclass "Integer", "A"
o = cl."instantiate"(5)
print o
print "\n"
.end

.namespace ["A"]
.sub __init method
.param int i
self = i
.end

> The code that backs perl classes can have implementations find_method
> that looks for __add methods with a given parameter signature, and the
> code that backs python classes can have implementations of find_metho
> that looks for __add__ methods with a different parameter signature.

Sam, there *should* be no difference between a pythonic "__add__"
and Parrot's "__add". Currently calling conventions don't match - I've
posted a mail WRT this and it needs fixing.

And Perl doesn't have an "__add" or such. Perl5 has "use overload '+' =>
..." Perl6 has a special infix:<+> syntax.

So the prerequisit is that Parrot's function signature of overloaded
methods is appropriate for Python and Perl.

We have:

  a = b + c

You already did show the Python code, which emits the Parrot opcode
"add", which does MMD dispatch. But we both posted an example that
overloads the operator "+" with a piece of user provided function.

You've stated that the code to provide this functionality is missing in
py*.pmc. But when you write it, you have of course to know, that a
setattribute (or a __dict__ update) with the key "__add__" is meaning an
overload of the "+" operator. Else you wouldn't be able to roll your own
dispatcher.

So when your translator encounters:

def myadd(self, r):
return self - r

class J(int):
__add__ = myadd

i = J(44)
print i, i + 2

you have to know that "myadd" will be the function that is called, when an
"add" operation for an object of class "J" is executed. This is the
place where you would call:

  Parrot_store_global(interp, "J", "__add", myadd);

The subroutine PMC myadd is stored into the namespace  of "J" as
the subroutine labeled "__add".

In the current system you would have to call "mmdvtregister" to
associate the "myadd" subroutine with the "add" opcode for the given
types - but that doesn't work for objects derived from class "J" nor for
mixed types.

So what I'm saying (or I'm trying to) is: when at the opcode level the
"add" opcode is actually acting as a method call, everything works as
now including dynamic overriding of MMD functions. And there is no need
at all for duplicating a lot of Parrot core functionality in the
py*.pmcs.

I've described that in "MMD dispatch". With the help of the inline cache
this scheme is 30% faster then the current static MMD table dispatch.

> - Sam Ruby

leo

Re: Python method overloading

[Sam Ruby]

Leopold Toetsch wrote:
Sam Ruby <[EMAIL PROTECTED]> wrote:
I continue to disagree with the part of this conclusion where you insert
find_method into the discussion.  To give a concrete example: at the
moment the lookup involved in abs_p_p does not involve the use of
find_method.
$ cat abs.imc
.sub main
.local pmc cl, o
cl = newclass "A"
$I0 = typeof cl
o = new $I0
$P0 = new Undef
$P0 = abs o
print $P0
.end
.namespace ["A"]
.sub __absolute
.param pmc d
d = "ok\n"
.end
$ parrot abs.imc
ok
$ parrot -t abs.imc
...
# find_method class 'A' method '__absolute': Sub
# Calling sub '__absolute'
...
But only for classes that inherit from delegate.
If Perl programmers need to know about Parrot method names, then
effectively this beomes part of the Perl language specification.  I do
not have the luxury of changing the Python language specification.
If you are targeting Parrot you have to know the opcode names *and* the
reserved method names, sorry.
People writing Python to Parrot translators need to know Parrot 
internals.  People who merely write in Python should not.

... (BTW, a combined instantiate method
does not map well to Python which has separate __new__ and __init__
methods.
We have the "__init" hook too. This is separate.
Ultimately, I believe that this will need to be revisited.
$ parrot -t abs.imc
...
6 new P17, I30 - P17=PMCNULL, I30=72
# find_method class 'A' method '__init': no
...  My recommendation
is to stick to primitives, and simply provide a new_p_p).
What is the second "_p" for?
What I am thinking of is something like:
  inline op new(out PMC) {
$1 = VTABLE_instantiate(interpreter, $2);
goto NEXT();
  }
Where instantiate does not support a variable number of arguments 
(Parrot calling conventions).  If you then want to init the PMC you have 
obtained, you can use the separate init hook (with a variable number of 
arguments).

That is only because the design you have in mind conflates Parrot and
language operations.  There is no reason that __abs__ couldn't call
VTABLE_abs, or that __add__ can't make use of MMD_ADD.
And if the class implements it's own "__absolute" or "__add", we do a
separate redispatch? And dynclasses/py* does it differently to
dynclasses/perl*. Why don't you just believe me that that's error prone
and slow ;-)
Only in the sense that CoRoutine and RetContinuation provide 
"incompatible" (i.e., different) implementations of invoke.  However, 
they are very compatible in the only sense that matters: they both 
implement the common protocol named "invoke".

Arguably, the very reason that a find_method VTABLE entry was provided 
was to enable different PMCs to provide different implementations of 
this protocol.

The code that backs perl classes can have implementations find_method 
that looks for __add methods with a given parameter signature, and the 
code that backs python classes can have implementations of find_metho 
that looks for __add__ methods with a different parameter signature.

- Sam Ruby

Re: Python method overloading

[Leopold Toetsch]

Sam Ruby <[EMAIL PROTECTED]> wrote:

> I continue to disagree with the part of this conclusion where you insert
> find_method into the discussion.  To give a concrete example: at the
> moment the lookup involved in abs_p_p does not involve the use of
> find_method.

$ cat abs.imc
.sub main
.local pmc cl, o
cl = newclass "A"
$I0 = typeof cl
o = new $I0
$P0 = new Undef
$P0 = abs o
print $P0
.end

.namespace ["A"]
.sub __absolute
.param pmc d
d = "ok\n"
.end

$ parrot abs.imc
ok

$ parrot -t abs.imc
...
# find_method class 'A' method '__absolute': Sub
# Calling sub '__absolute'
...

> ... Nor does the lookup involved in find_method_p_p_s for that
> matter.

???

> If Perl programmers need to know about Parrot method names, then
> effectively this beomes part of the Perl language specification.  I do
> not have the luxury of changing the Python language specification.

If you are targeting Parrot you have to know the opcode names *and* the
reserved method names, sorry.

> I continue to push for the "namespace" for Parrot internal dispatching
> and language level method names to be disjoint.

That's a different thing that we can consider.

> ... If Python metaclasses
> get surprising results if they happen to define a method named
> "instantiate", expect a bug report.

Ok. The call syntax should rather be "__instantiate" to be consistent.

> ... (BTW, a combined instantiate method
> does not map well to Python which has separate __new__ and __init__
> methods.

We have the "__init" hook too. This is separate.

$ parrot -t abs.imc
...
6 new P17, I30 - P17=PMCNULL, I30=72
# find_method class 'A' method '__init': no

> ...  My recommendation
> is to stick to primitives, and simply provide a new_p_p).

What is the second "_p" for?

> There needs to be separate "find_method" operations for external (i.e.,
> visible at the language level) names and Parrot internal names.

Maybe. But why is it necessary?

> That is only because the design you have in mind conflates Parrot and
> language operations.  There is no reason that __abs__ couldn't call
> VTABLE_abs, or that __add__ can't make use of MMD_ADD.

And if the class implements it's own "__absolute" or "__add", we do a
separate redispatch? And dynclasses/py* does it differently to
dynclasses/perl*. Why don't you just believe me that that's error prone
and slow ;-)

> - Sam Ruby

leo

Re: Python method overloading

[Sam Ruby]

Leopold Toetsch wrote:

Leo - at one point you indicated that you might be interested in helping
to factor out the common code again.
Sure, and I'm not stopping that. The overall conclusion of (even infix
operator) method lookup was that it has to be done at runtime. It is up to
the PMCs to provide a proper and language-specific find_method to locate
the involved operation, MMD or not.
I continue to disagree with the part of this conclusion where you insert 
find_method into the discussion.  To give a concrete example: at the 
moment the lookup involved in abs_p_p does not involve the use of 
find_method.  Nor does the lookup involved in find_method_p_p_s for that 
matter.

If Perl programmers need to know about Parrot method names, then 
effectively this beomes part of the Perl language specification.  I do 
not have the luxury of changing the Python language specification.

I continue to push for the "namespace" for Parrot internal dispatching 
and language level method names to be disjoint.  If Python metaclasses 
get surprising results if they happen to define a method named 
"instantiate", expect a bug report.  (BTW, a combined instantiate method 
does not map well to Python which has separate __new__ and __init__ 
methods.  You are going to find issues like these every time you try to 
put object oriented semantics into the Parrot core.  My recommendation 
is to stick to primitives, and simply provide a new_p_p).

There needs to be separate "find_method" operations for external (i.e., 
visible at the language level) names and Parrot internal names.

And the runcore is responsible for calling the method. While you seem to
admit that it has to be done  at runtime you are doubting that it can be
done fast and you are rolling your own incompatible dispatch scheme (how
should that be faster then?). This made me utter above sentence.
"as long as we have a proper protocol that everyone can conform to, we 
should be OK." - Dan Sugalski, 2004/11/29

Further: The function signature of overloaded infix operations (and
maybe others) is currently not appropriate for Python (and likely other
languages).
That is only because the design you have in mind conflates Parrot and 
language operations.  There is no reason that __abs__ couldn't call 
VTABLE_abs, or that __add__ can't make use of MMD_ADD.

- Sam Ruby

Re: Python method overloading

[Leopold Toetsch]

Sam Ruby <[EMAIL PROTECTED]> wrote:
> Leopold Toetsch wrote:

>> Sam Ruby <[EMAIL PROTECTED]> wrote:
>>
>> [ snipped - all ok }
>>
>>>If I define an __add__ method with 16 arguments, Python will not throw
>>>an exception.
>>
>> I didn't write that. I've said: *if* you call it via "a + b", Python
>> throws an exception - that one I've shown. Anyway...

> What you wrote (and snipped) was

> "I'd say, if you define an '__add__' method with 16 arguments, Python
> will throw an exception,..."

> To which I responded with the above.

and after the snipped ellipsis my sentence did continue with

" if you try to use C<+> with an object of that class:"

>> You are still not getting the principal of the scheme, IMHO. It has
>> nothing to do with Perl6 or any other language, nor with Python.

> Either that, or I *am* getting the principle of the scheme.  I guess
> that this is the point where I need to return back to writing code and
> test cases.

> Leo - at one point you indicated that you might be interested in helping
> to factor out the common code again.

Sure, and I'm not stopping that. The overall conclusion of (even infix
operator) method lookup was that it has to be done at runtime. It is up to
the PMCs to provide a proper and language-specific find_method to locate
the involved operation, MMD or not.

And the runcore is responsible for calling the method. While you seem to
admit that it has to be done  at runtime you are doubting that it can be
done fast and you are rolling your own incompatible dispatch scheme (how
should that be faster then?). This made me utter above sentence.

Further: The function signature of overloaded infix operations (and
maybe others) is currently not appropriate for Python (and likely other
languages).

> - Sam Ruby

leo

Re: Python method overloading

[Sam Ruby]

Leopold Toetsch wrote:
Sam Ruby <[EMAIL PROTECTED]> wrote:
[ snipped - all ok }
If I define an __add__ method with 16 arguments, Python will not throw
an exception.
I didn't write that. I've said: *if* you call it via "a + b", Python
throws an exception - that one I've shown. Anyway...
What you wrote (and snipped) was
"I'd say, if you define an '__add__' method with 16 arguments, Python
will throw an exception,..."
To which I responded with the above.
You are still not getting the principal of the scheme, IMHO. It has
nothing to do with Perl6 or any other language, nor with Python.
Either that, or I *am* getting the principle of the scheme.  I guess 
that this is the point where I need to return back to writing code and 
test cases.

Leo - at one point you indicated that you might be interested in helping 
to factor out the common code again.  Please feel free to do so whenever 
you are ready.  All I ask is that you don't break the test cases.

- Sam Ruby
P.S.  No fair changing the test cases either.  ;-)

Re: Python method overloading

[Leopold Toetsch]

Sam Ruby <[EMAIL PROTECTED]> wrote:

[ snipped - all ok }

> If I define an __add__ method with 16 arguments, Python will not throw
> an exception.

I didn't write that. I've said: *if* you call it via "a + b", Python
throws an exception - that one I've shown. Anyway...

> If this is done at runtime, the it need not be done at compile time.

... Yes. That's the overall conclusiom it seems. It can be done partially
at compile time, and it isn't worth the effort to try it, because
languages we are targeting are too dynamic.

> However, it doesn't stop here.  Just like methods can be added
> dynamically by name at runtime, they can be accessed dynamically by
> name.  That means that all method lookups will need to be preceeded by a
> hash lookup.  An not just on Python objects, but *all* objects.

... preceeded by some kind of lookup, which is defined by
"class->vtable->find_method()" of the responsible metaclass.  Being it
one or 100 hash lookups in properties, dicts, globals and what not. It
doesn't matter. Dot.

> That's why I object to characterizations like "dynamic dispatch is 30%
> faster than...".  What will ultimately result if it is mandated that all
> languages adopt Perl6's semantics is that an ADDITIONAL dynamic dispatch
> will be required to make non-Perl6 functions work.

You are still not getting the principal of the scheme, IMHO. It has
nothing to do with Perl6 or any other language, nor with Python.

The original subject: "premature pessimization" strikes back :)

Whe just do a dynamic lookup at runtime - that's all.

The e.g. "add" opcode calls left->vtable->find_method(), and probably
more if the return results inidicates MMD. Eventually one of the
find_method calls returns a function that does implement the "__add"
method for the involved types. Or a (possibly user provided) distance
function decides, which function to call. It doesn't matter.

Then the *runcore* calls the function and *caches* the function pointer.

Next time the call is instantaneous, given that the language is able to
call a cache invalidation function, if method lookup order (for that
class) changes.

The call to the invalidation function is possible, even for Python.
*Iff* you can roll your own method dispatch, you eventually need to
know, which method you call. That has to be defined. You can as well
call a cache invalidation function, if something changes here (I hope)

> PerlScalar's implementation of the add will know about how to implement
> Perl 6's multi sub *infix.  PyObject won't, but it will know about
> Python's __meta__ and __init_class__.

If even an "add" instruction doesn't work outside of one HLL, we can
forget any interoperbility. "__meta__" and what not Python semantics can
be added - or not :-) But let's first concentrate on the basics.

> - Sam Ruby

leo

Re: Python method overloading

[Sam Ruby]

Leopold Toetsch wrote:
Sam Ruby <[EMAIL PROTECTED]> wrote:
Leopold Toetsch wrote:

Here's the part that you snipped that addresses that question:

  > And there is a piece that I haven't written yet that will do the
  > reverse: if MMD_ADD is called on a PyObject that has not provided
  > such behavior, then an any __add__ method provided needs to be
  > called.
Ok. But that would imply that HLL interoperbility isn't really possible. Or
just at a minimal surface level. But see below.
I don't believe that to be the case.  If a Perl subroutine were to call 
a Python function and pass a PerlInt as a parameter, the receiving 
function should expect to be able to do addition via tha "+" operator, 
but should not be expect to find an "__add__" method on such objects. 
Instead, and if they cared to, they could explicitly call the "__add" 
method provided.

The reverse should also be true, if Python function were to call a Perl 
subroutine and pass a PyInt as a parameter, the receiving subroutine 
should expect to be able to do addition via the "+" operator, but not 
expect to find an "__add" method on such objects.  Instead, and if they 
cared to, they could explicitly call the "__add__" method provided.

I would consider that significant interoperability with only minimal 
restrictions.

While the Python people aren't stopping to talk about the clearness of
their language, nothing is clear and explicit, when it comes to overloading
or metaclasses.
Please don't do that.  I am not trying to extoll the virtues of Python, 
merely trying to implement it.

Anyway, IMHO, "class.__add__ = foo" or your example manipulating
"class.__dict__" (another special attribute name!) is the point, where
you can install Parrot semantics WRT method overloading.
Hold that thought.  I'll answer this below.
Now, given the above sample, let's revisit the statement that "The
Python translator needs just a translation table for the common core
methods."
We both know that's a simplification :) You've to install the methods of
course ...
Again, it can't be done exclusively at translation time.  It needs to be 
done at runtime.  And if it is done at runtime, it need not be done at 
translation time at all.  More below.

How, exactly, would that be done?  Given that the method name is simply
a string... used as a key in dictionary... with a different parameter
signature than the hypothetical Parrot __add method.
The "class.__dict__" dictionary is special. Setting an "__add__" key too.
The combined meaning is overloading. The different signature is a
problem, yes - I've already mentioned that. And Parrot's "__add" method
is not hypothetical :-)
$ grep __add t/pmc/object*.t
Here I'll apologize for being unclear.  Yes, there is code in the 
existing object class in support of Perl's S06.  What's hypothetical is 
the presumption that all languages will adopt Perl 6's naming convention 
for methods.

That's why I say:

In the general case, looking for "reserved" method names at "compile"
time doesn't work.

"__add__" is reserved in Python and corresponds directly to "__add" in
Parrot. I don't think that doesn't work.

__add__ is *not* reserved in Python.
Does it matter if the name is actually reserved? The meaning is
important.
It does matter.  Python classes are dictionaries of objects, some of 
which may be functions.  You may extract objects from that dictionary 
and access them later.  The "meaning" in such a scenario is not apparent 
until well after all interaction with the compile and runtime 
dictionaries is over.

... There just is some syntatic sugar
that provide a shorthand for certain signatures.  I am free to define
__add__ methods that have zero or sixteen arguments.  I won't be able to
call such methods with the convenient shorthand, but other than that,
they should work.
I'd say, if you define an '__add__' method with 16 arguments, Python
will throw an exception, if you try to use C<+> with an object of that
class:
If I define an __add__ method with 16 arguments, Python will not throw 
an exception.

I've already shown that it's possible to go with fully dynamic dispatch
*and* 30% faster for MMD and 70% faster for overloaded operations. First
correct and complete, then speed considerations.

Neither of which match Python semantics.  We are going to need a system
where classes are anonymous, not global.
Why? And how do you find your class then:
  c = C()
 ...
  3  22 LOAD_NAME1 (C)
 25 CALL_FUNCTION0
$pirate -d c.py
...
find_lex $P0, 'C'
$P1=$P0()
store_lex -1, 'c', $P1
The important part isn't simply in which hash a given class name is 
looked up in, but that classes themselves in Python are transient 
objects subject to garbage collection.

... Where methods are properties
that can be added simply by calling the equivalent of set_pmc_keyed.
Nah. Methods aren't properties, but ...
No?  Try the following:
   x = "abcdef".find
   print x('c')
The "set_pmc_keye

Python method overloading (was: Premature pessimization)

[Leopold Toetsch]

Sam Ruby <[EMAIL PROTECTED]> wrote:
> Leopold Toetsch wrote:

> Here's the part that you snipped that addresses that question:

>> And there is a piece that I haven't written yet that will do the
>> reverse: if MMD_ADD is called on a PyObject that has not provided
>> such behavior, then an any __add__ method provided needs to be
>> called.

Ok. But that would imply that HLL interoperbility isn't really possible. Or
just at a minimal surface level. But see below.

> Since you provided an Evil Leo sample, let me provide an Evil Sam sample:

>d = {
>  "__init__": lambda self,x: setattr(self, "value", x),
>  "__add__":  lambda self,x: str(self.value) + str(x.value)
>}

>def dict2class(d):
>  class c: pass
>  c.__dict__.update(d)
   ^^^

This is the critical part of it. The "__dict__" of your class provides the
namespace. Setting a key in that namespace (or an attribute of your class
with that key) has a special meaning in Python, *if* that key happens to
be one of the method names.

While the Python people aren't stopping to talk about the clearness of
their language, nothing is clear and explicit, when it comes to overloading
or metaclasses.

Anyway, IMHO, "class.__add__ = foo" or your example manipulating
"class.__dict__" (another special attribute name!) is the point, where
you can install Parrot semantics WRT method overloading.

> Now, given the above sample, let's revisit the statement that "The
> Python translator needs just a translation table for the common core
> methods."

We both know that's a simplification :) You've to install the methods of
course ...

> How, exactly, would that be done?  Given that the method name is simply
> a string... used as a key in dictionary... with a different parameter
> signature than the hypothetical Parrot __add method.

The "class.__dict__" dictionary is special. Setting an "__add__" key too.
The combined meaning is overloading. The different signature is a
problem, yes - I've already mentioned that. And Parrot's "__add" method
is not hypothetical :-)

$ grep __add t/pmc/object*.t

> That's why I say:

>>>In the general case, looking for "reserved" method names at "compile"
>>>time doesn't work.

>> "__add__" is reserved in Python and corresponds directly to "__add" in
>> Parrot. I don't think that doesn't work.

> __add__ is *not* reserved in Python.

Does it matter if the name is actually reserved? The meaning is
important.

> ... There just is some syntatic sugar
> that provide a shorthand for certain signatures.  I am free to define
> __add__ methods that have zero or sixteen arguments.  I won't be able to
> call such methods with the convenient shorthand, but other than that,
> they should work.

I'd say, if you define an '__add__' method with 16 arguments, Python
will throw an exception, if you try to use C<+> with an object of that
class:

  TypeError: myadd() takes exactly 16 arguments (2 given)

So that's rather hypothetical. And if you always use x."__add__"(16
args) Parrot will just run the function.

>>>I personally don't think that performance considerations should be out
>>>of bounds in these discussions
>>
>> I've already shown that it's possible to go with fully dynamic dispatch
>> *and* 30% faster for MMD and 70% faster for overloaded operations. First
>> correct and complete, then speed considerations.

> Neither of which match Python semantics.  We are going to need a system
> where classes are anonymous, not global.

Why? And how do you find your class then:

  c = C()
 ...
  3  22 LOAD_NAME1 (C)
 25 CALL_FUNCTION0

> ... Where methods are properties
> that can be added simply by calling the equivalent of set_pmc_keyed.

Nah. Methods aren't properties, but ...

The "set_pmc_keyed" on "__dict__" (or an equivalent setattribute call)
of your type system is responsible to create Parrot semantics for method
calls :-)

> - Sam Ruby

leo