Re: Python method overloading

[Leopold Toetsch]

Sam Ruby [EMAIL PROTECTED] wrote:
 Leopold Toetsch wrote:

 # find_method class 'A' method '__absolute': Sub
 # Calling sub '__absolute'

 But only for classes that inherit from delegate.

Yes of course. Objects's derived from ParrotObject (i.e. Parrot standard
objects) dispatch to overloaded functions via the meta-class helper
class delegate. Standard PMCs call pmc-vtable-absolute().

But infix operations like the add opcode dispatch via mmd_dispatch_*
and have no delegate functionality.

I've posted a graphic of all these possibilities in one of the mails.

This is the reason why I'm prefering to replace all with just one
scheme that works, i.e. doing full dynamic dispatch at the opcode level.

 People writing Python to Parrot translators need to know Parrot
 internals.  People who merely write in Python should not.

Why on earth should Python writers know Parrot internals? Why do you
come to such a conclusion?

... (BTW, a combined instantiate method
does not map well to Python which has separate __new__ and __init__
methods.

 We have the __init hook too. This is separate.

 Ultimately, I believe that this will need to be revisited.

What is wrong? Does it not work?

...  My recommendation
is to stick to primitives, and simply provide a new_p_p).

 What is the second _p for?

 What I am thinking of is something like:

inline op new(out PMC) {
  $1 = VTABLE_instantiate(interpreter, $2);

There is no second argument on the new signature. But I presume it's a
class PMC. Well doing:

  cl = getclass Foo
  o = cl.instantiate()

does exactly that - it constructs a new Integer with the default value by
calling pmc_new/init.

OTOH when you have

  cl = getclass Foo
  o = cl.instantiate(5)

you have two possiblities: implement the instantiate vtable (or let a
user provided overridden method do the initialization) or do the init
stuff in __init, which get's again the arguments according to pdd03.

   If you then want to init the PMC you have
 obtained, you can use the separate init hook (with a variable number of
 arguments).

This is working already.

.sub main
.local pmc cl, o
cl = subclass Integer, A
o = cl.instantiate(5)
print o
print \n
.end

.namespace [A]
.sub __init method
.param int i
self = i
.end

 The code that backs perl classes can have implementations find_method
 that looks for __add methods with a given parameter signature, and the
 code that backs python classes can have implementations of find_metho
 that looks for __add__ methods with a different parameter signature.

Sam, there *should* be no difference between a pythonic __add__
and Parrot's __add. Currently calling conventions don't match - I've
posted a mail WRT this and it needs fixing.

And Perl doesn't have an __add or such. Perl5 has use overload '+' =
... Perl6 has a special infix:+ syntax.

So the prerequisit is that Parrot's function signature of overloaded
methods is appropriate for Python and Perl.

We have:

  a = b + c

You already did show the Python code, which emits the Parrot opcode
add, which does MMD dispatch. But we both posted an example that
overloads the operator + with a piece of user provided function.

You've stated that the code to provide this functionality is missing in
py*.pmc. But when you write it, you have of course to know, that a
setattribute (or a __dict__ update) with the key __add__ is meaning an
overload of the + operator. Else you wouldn't be able to roll your own
dispatcher.

So when your translator encounters:

def myadd(self, r):
return self - r

class J(int):
__add__ = myadd

i = J(44)
print i, i + 2

you have to know that myadd will be the function that is called, when an
add operation for an object of class J is executed. This is the
place where you would call:

  Parrot_store_global(interp, J, __add, myadd);

The subroutine PMC myadd is stored into the namespace  of J as
the subroutine labeled __add.

In the current system you would have to call mmdvtregister to
associate the myadd subroutine with the add opcode for the given
types - but that doesn't work for objects derived from class J nor for
mixed types.

So what I'm saying (or I'm trying to) is: when at the opcode level the
add opcode is actually acting as a method call, everything works as
now including dynamic overriding of MMD functions. And there is no need
at all for duplicating a lot of Parrot core functionality in the
py*.pmcs.

I've described that in MMD dispatch. With the help of the inline cache
this scheme is 30% faster then the current static MMD table dispatch.

 - Sam Ruby

leo

Re: Python method overloading

[Sam Ruby]

Leopold Toetsch wrote:
Sam Ruby [EMAIL PROTECTED] wrote:
I continue to disagree with the part of this conclusion where you insert
find_method into the discussion.  To give a concrete example: at the
moment the lookup involved in abs_p_p does not involve the use of
find_method.
$ cat abs.imc
.sub main
.local pmc cl, o
cl = newclass A
$I0 = typeof cl
o = new $I0
$P0 = new Undef
$P0 = abs o
print $P0
.end
.namespace [A]
.sub __absolute
.param pmc d
d = ok\n
.end
$ parrot abs.imc
ok
$ parrot -t abs.imc
...
# find_method class 'A' method '__absolute': Sub
# Calling sub '__absolute'
...
But only for classes that inherit from delegate.
If Perl programmers need to know about Parrot method names, then
effectively this beomes part of the Perl language specification.  I do
not have the luxury of changing the Python language specification.
If you are targeting Parrot you have to know the opcode names *and* the
reserved method names, sorry.
People writing Python to Parrot translators need to know Parrot 
internals.  People who merely write in Python should not.

... (BTW, a combined instantiate method
does not map well to Python which has separate __new__ and __init__
methods.
We have the __init hook too. This is separate.
Ultimately, I believe that this will need to be revisited.
$ parrot -t abs.imc
...
6 new P17, I30 - P17=PMCNULL, I30=72
# find_method class 'A' method '__init': no
...  My recommendation
is to stick to primitives, and simply provide a new_p_p).
What is the second _p for?
What I am thinking of is something like:
  inline op new(out PMC) {
$1 = VTABLE_instantiate(interpreter, $2);
goto NEXT();
  }
Where instantiate does not support a variable number of arguments 
(Parrot calling conventions).  If you then want to init the PMC you have 
obtained, you can use the separate init hook (with a variable number of 
arguments).

That is only because the design you have in mind conflates Parrot and
language operations.  There is no reason that __abs__ couldn't call
VTABLE_abs, or that __add__ can't make use of MMD_ADD.
And if the class implements it's own __absolute or __add, we do a
separate redispatch? And dynclasses/py* does it differently to
dynclasses/perl*. Why don't you just believe me that that's error prone
and slow ;-)
Only in the sense that CoRoutine and RetContinuation provide 
incompatible (i.e., different) implementations of invoke.  However, 
they are very compatible in the only sense that matters: they both 
implement the common protocol named invoke.

Arguably, the very reason that a find_method VTABLE entry was provided 
was to enable different PMCs to provide different implementations of 
this protocol.

The code that backs perl classes can have implementations find_method 
that looks for __add methods with a given parameter signature, and the 
code that backs python classes can have implementations of find_metho 
that looks for __add__ methods with a different parameter signature.

- Sam Ruby

Re: Python method overloading

[Leopold Toetsch]

Sam Ruby [EMAIL PROTECTED] wrote:
 Leopold Toetsch wrote:

 Sam Ruby [EMAIL PROTECTED] wrote:

 [ snipped - all ok }

If I define an __add__ method with 16 arguments, Python will not throw
an exception.

 I didn't write that. I've said: *if* you call it via a + b, Python
 throws an exception - that one I've shown. Anyway...

 What you wrote (and snipped) was

 I'd say, if you define an '__add__' method with 16 arguments, Python
 will throw an exception,...

 To which I responded with the above.

and after the snipped ellipsis my sentence did continue with

 if you try to use C+ with an object of that class:

 You are still not getting the principal of the scheme, IMHO. It has
 nothing to do with Perl6 or any other language, nor with Python.

 Either that, or I *am* getting the principle of the scheme.  I guess
 that this is the point where I need to return back to writing code and
 test cases.

 Leo - at one point you indicated that you might be interested in helping
 to factor out the common code again.

Sure, and I'm not stopping that. The overall conclusion of (even infix
operator) method lookup was that it has to be done at runtime. It is up to
the PMCs to provide a proper and language-specific find_method to locate
the involved operation, MMD or not.

And the runcore is responsible for calling the method. While you seem to
admit that it has to be done  at runtime you are doubting that it can be
done fast and you are rolling your own incompatible dispatch scheme (how
should that be faster then?). This made me utter above sentence.

Further: The function signature of overloaded infix operations (and
maybe others) is currently not appropriate for Python (and likely other
languages).

 - Sam Ruby

leo

Re: Python method overloading

[Sam Ruby]

Leopold Toetsch wrote:

Leo - at one point you indicated that you might be interested in helping
to factor out the common code again.
Sure, and I'm not stopping that. The overall conclusion of (even infix
operator) method lookup was that it has to be done at runtime. It is up to
the PMCs to provide a proper and language-specific find_method to locate
the involved operation, MMD or not.
I continue to disagree with the part of this conclusion where you insert 
find_method into the discussion.  To give a concrete example: at the 
moment the lookup involved in abs_p_p does not involve the use of 
find_method.  Nor does the lookup involved in find_method_p_p_s for that 
matter.

If Perl programmers need to know about Parrot method names, then 
effectively this beomes part of the Perl language specification.  I do 
not have the luxury of changing the Python language specification.

I continue to push for the namespace for Parrot internal dispatching 
and language level method names to be disjoint.  If Python metaclasses 
get surprising results if they happen to define a method named 
instantiate, expect a bug report.  (BTW, a combined instantiate method 
does not map well to Python which has separate __new__ and __init__ 
methods.  You are going to find issues like these every time you try to 
put object oriented semantics into the Parrot core.  My recommendation 
is to stick to primitives, and simply provide a new_p_p).

There needs to be separate find_method operations for external (i.e., 
visible at the language level) names and Parrot internal names.

And the runcore is responsible for calling the method. While you seem to
admit that it has to be done  at runtime you are doubting that it can be
done fast and you are rolling your own incompatible dispatch scheme (how
should that be faster then?). This made me utter above sentence.
as long as we have a proper protocol that everyone can conform to, we 
should be OK. - Dan Sugalski, 2004/11/29

Further: The function signature of overloaded infix operations (and
maybe others) is currently not appropriate for Python (and likely other
languages).
That is only because the design you have in mind conflates Parrot and 
language operations.  There is no reason that __abs__ couldn't call 
VTABLE_abs, or that __add__ can't make use of MMD_ADD.

- Sam Ruby

Re: Python method overloading

[Leopold Toetsch]

Sam Ruby [EMAIL PROTECTED] wrote:

 I continue to disagree with the part of this conclusion where you insert
 find_method into the discussion.  To give a concrete example: at the
 moment the lookup involved in abs_p_p does not involve the use of
 find_method.

$ cat abs.imc
.sub main
.local pmc cl, o
cl = newclass A
$I0 = typeof cl
o = new $I0
$P0 = new Undef
$P0 = abs o
print $P0
.end

.namespace [A]
.sub __absolute
.param pmc d
d = ok\n
.end

$ parrot abs.imc
ok

$ parrot -t abs.imc
...
# find_method class 'A' method '__absolute': Sub
# Calling sub '__absolute'
...

 ... Nor does the lookup involved in find_method_p_p_s for that
 matter.

???

 If Perl programmers need to know about Parrot method names, then
 effectively this beomes part of the Perl language specification.  I do
 not have the luxury of changing the Python language specification.

If you are targeting Parrot you have to know the opcode names *and* the
reserved method names, sorry.

 I continue to push for the namespace for Parrot internal dispatching
 and language level method names to be disjoint.

That's a different thing that we can consider.

 ... If Python metaclasses
 get surprising results if they happen to define a method named
 instantiate, expect a bug report.

Ok. The call syntax should rather be __instantiate to be consistent.

 ... (BTW, a combined instantiate method
 does not map well to Python which has separate __new__ and __init__
 methods.

We have the __init hook too. This is separate.

$ parrot -t abs.imc
...
6 new P17, I30 - P17=PMCNULL, I30=72
# find_method class 'A' method '__init': no

 ...  My recommendation
 is to stick to primitives, and simply provide a new_p_p).

What is the second _p for?

 There needs to be separate find_method operations for external (i.e.,
 visible at the language level) names and Parrot internal names.

Maybe. But why is it necessary?

 That is only because the design you have in mind conflates Parrot and
 language operations.  There is no reason that __abs__ couldn't call
 VTABLE_abs, or that __add__ can't make use of MMD_ADD.

And if the class implements it's own __absolute or __add, we do a
separate redispatch? And dynclasses/py* does it differently to
dynclasses/perl*. Why don't you just believe me that that's error prone
and slow ;-)

 - Sam Ruby

leo

Re: Python method overloading

[Sam Ruby]

Leopold Toetsch wrote:
Sam Ruby [EMAIL PROTECTED] wrote:
Leopold Toetsch wrote:

Here's the part that you snipped that addresses that question:

   And there is a piece that I haven't written yet that will do the
   reverse: if MMD_ADD is called on a PyObject that has not provided
   such behavior, then an any __add__ method provided needs to be
   called.
Ok. But that would imply that HLL interoperbility isn't really possible. Or
just at a minimal surface level. But see below.
I don't believe that to be the case.  If a Perl subroutine were to call 
a Python function and pass a PerlInt as a parameter, the receiving 
function should expect to be able to do addition via tha + operator, 
but should not be expect to find an __add__ method on such objects. 
Instead, and if they cared to, they could explicitly call the __add 
method provided.

The reverse should also be true, if Python function were to call a Perl 
subroutine and pass a PyInt as a parameter, the receiving subroutine 
should expect to be able to do addition via the + operator, but not 
expect to find an __add method on such objects.  Instead, and if they 
cared to, they could explicitly call the __add__ method provided.

I would consider that significant interoperability with only minimal 
restrictions.

While the Python people aren't stopping to talk about the clearness of
their language, nothing is clear and explicit, when it comes to overloading
or metaclasses.
Please don't do that.  I am not trying to extoll the virtues of Python, 
merely trying to implement it.

Anyway, IMHO, class.__add__ = foo or your example manipulating
class.__dict__ (another special attribute name!) is the point, where
you can install Parrot semantics WRT method overloading.
Hold that thought.  I'll answer this below.
Now, given the above sample, let's revisit the statement that The
Python translator needs just a translation table for the common core
methods.
We both know that's a simplification :) You've to install the methods of
course ...
Again, it can't be done exclusively at translation time.  It needs to be 
done at runtime.  And if it is done at runtime, it need not be done at 
translation time at all.  More below.

How, exactly, would that be done?  Given that the method name is simply
a string... used as a key in dictionary... with a different parameter
signature than the hypothetical Parrot __add method.
The class.__dict__ dictionary is special. Setting an __add__ key too.
The combined meaning is overloading. The different signature is a
problem, yes - I've already mentioned that. And Parrot's __add method
is not hypothetical :-)
$ grep __add t/pmc/object*.t
Here I'll apologize for being unclear.  Yes, there is code in the 
existing object class in support of Perl's S06.  What's hypothetical is 
the presumption that all languages will adopt Perl 6's naming convention 
for methods.

That's why I say:

In the general case, looking for reserved method names at compile
time doesn't work.

__add__ is reserved in Python and corresponds directly to __add in
Parrot. I don't think that doesn't work.

__add__ is *not* reserved in Python.
Does it matter if the name is actually reserved? The meaning is
important.
It does matter.  Python classes are dictionaries of objects, some of 
which may be functions.  You may extract objects from that dictionary 
and access them later.  The meaning in such a scenario is not apparent 
until well after all interaction with the compile and runtime 
dictionaries is over.

... There just is some syntatic sugar
that provide a shorthand for certain signatures.  I am free to define
__add__ methods that have zero or sixteen arguments.  I won't be able to
call such methods with the convenient shorthand, but other than that,
they should work.
I'd say, if you define an '__add__' method with 16 arguments, Python
will throw an exception, if you try to use C+ with an object of that
class:
If I define an __add__ method with 16 arguments, Python will not throw 
an exception.

I've already shown that it's possible to go with fully dynamic dispatch
*and* 30% faster for MMD and 70% faster for overloaded operations. First
correct and complete, then speed considerations.

Neither of which match Python semantics.  We are going to need a system
where classes are anonymous, not global.
Why? And how do you find your class then:
  c = C()
 ...
  3  22 LOAD_NAME1 (C)
 25 CALL_FUNCTION0
$pirate -d c.py
...
find_lex $P0, 'C'
$P1=$P0()
store_lex -1, 'c', $P1
The important part isn't simply in which hash a given class name is 
looked up in, but that classes themselves in Python are transient 
objects subject to garbage collection.

... Where methods are properties
that can be added simply by calling the equivalent of set_pmc_keyed.
Nah. Methods aren't properties, but ...
No?  Try the following:
   x = abcdef.find
   print x('c')
The set_pmc_keyed on __dict__ (or an equivalent setattribute 

Re: Python method overloading

[Leopold Toetsch]

Sam Ruby [EMAIL PROTECTED] wrote:

[ snipped - all ok }

 If I define an __add__ method with 16 arguments, Python will not throw
 an exception.

I didn't write that. I've said: *if* you call it via a + b, Python
throws an exception - that one I've shown. Anyway...

 If this is done at runtime, the it need not be done at compile time.

... Yes. That's the overall conclusiom it seems. It can be done partially
at compile time, and it isn't worth the effort to try it, because
languages we are targeting are too dynamic.

 However, it doesn't stop here.  Just like methods can be added
 dynamically by name at runtime, they can be accessed dynamically by
 name.  That means that all method lookups will need to be preceeded by a
 hash lookup.  An not just on Python objects, but *all* objects.

... preceeded by some kind of lookup, which is defined by
class-vtable-find_method() of the responsible metaclass.  Being it
one or 100 hash lookups in properties, dicts, globals and what not. It
doesn't matter. Dot.

 That's why I object to characterizations like dynamic dispatch is 30%
 faster than  What will ultimately result if it is mandated that all
 languages adopt Perl6's semantics is that an ADDITIONAL dynamic dispatch
 will be required to make non-Perl6 functions work.

You are still not getting the principal of the scheme, IMHO. It has
nothing to do with Perl6 or any other language, nor with Python.

The original subject: premature pessimization strikes back :)

Whe just do a dynamic lookup at runtime - that's all.

The e.g. add opcode calls left-vtable-find_method(), and probably
more if the return results inidicates MMD. Eventually one of the
find_method calls returns a function that does implement the __add
method for the involved types. Or a (possibly user provided) distance
function decides, which function to call. It doesn't matter.

Then the *runcore* calls the function and *caches* the function pointer.

Next time the call is instantaneous, given that the language is able to
call a cache invalidation function, if method lookup order (for that
class) changes.

The call to the invalidation function is possible, even for Python.
*Iff* you can roll your own method dispatch, you eventually need to
know, which method you call. That has to be defined. You can as well
call a cache invalidation function, if something changes here (I hope)

 PerlScalar's implementation of the add will know about how to implement
 Perl 6's multi sub *infix.  PyObject won't, but it will know about
 Python's __meta__ and __init_class__.

If even an add instruction doesn't work outside of one HLL, we can
forget any interoperbility. __meta__ and what not Python semantics can
be added - or not :-) But let's first concentrate on the basics.

 - Sam Ruby

leo

Re: Python method overloading

[Sam Ruby]

Leopold Toetsch wrote:
Sam Ruby [EMAIL PROTECTED] wrote:
[ snipped - all ok }
If I define an __add__ method with 16 arguments, Python will not throw
an exception.
I didn't write that. I've said: *if* you call it via a + b, Python
throws an exception - that one I've shown. Anyway...
What you wrote (and snipped) was
I'd say, if you define an '__add__' method with 16 arguments, Python
will throw an exception,...
To which I responded with the above.
You are still not getting the principal of the scheme, IMHO. It has
nothing to do with Perl6 or any other language, nor with Python.
Either that, or I *am* getting the principle of the scheme.  I guess 
that this is the point where I need to return back to writing code and 
test cases.

Leo - at one point you indicated that you might be interested in helping 
to factor out the common code again.  Please feel free to do so whenever 
you are ready.  All I ask is that you don't break the test cases.

- Sam Ruby
P.S.  No fair changing the test cases either.  ;-)