Hi Rony,

I showed you some code samples in Java language, so I had to explain what java compiler is doing too, for you to fully understand what's going on. You see, when using Java and HM.invokeWithArguments(), conversions (varargs included) are taking place at two levels. The 1st conversion is performed by javac generated bytecode (when invoking .invokeWithArguments, which is a varargs method) and the 2nd is the conversion performed by the HM.invokeWithArguments() method itself when it invokes the target method. The 1st is specified by JLS, while the 2nd is specified in the javadoc of .invokeWithArguments.

If we skip the 1st conversion, since you probably know what compiler is doing, you still have to be aware of the 2nd conversion, performed by MH.invokeWithArguments() (and MH.invoke()). First thing you have to know is the following:

/Method handles produced by lookups or constant loads from methods or constructors with the variable arity modifier bit (0x0080) have a corresponding variable arity, as if they were defined with the help of asVarargsCollector.//
This part does not mention the MethodHandles.Lookup.unreflect(Method) way of producing MethodHandle(s), but its all the same. You have to be aware that some method handles returned to you have a special property: they are varargs collectors. When invoking those methods, they accept any number of trailing positional arguments and collect them into an array argument.

This does not happen when you invoke such methods using reflection.

So if you want HM.invokeWithArguments() to behave the same as Method.invoke(), you have to transform the method handles obtained from lookup / unreflect with MH.asFixedArity() method. If the method handle is already a fixed arity method handle, this method will just return it, else it will return the equivalent method handle that doesn't perform collection of trailing positional arguments into an array argument. Using MH.asFixedArity() on each and every method handle you obtain by unreflect should give you the same behavior of MH.invokeWithArguments() as using Method.invoke().

Regards, Peter

On 06/10/18 14:16, Rony G. Flatscher wrote:

Dear Peter:

thank you very much for your thorough and extensive analysis taking the evolvement of Java into account!

You use the Java compiler and its behaviour (which adds syntax sugar to the Java language and is known to stick strictly to the Java language specification) for explanations and as reference, which is also what I have been doing in the past to counter-check the implemented behaviour of the bridge.

The Java compiler has a very different outset: *at compilation time* it knows everything it needs to know for compiling. If something is missing or wrong at compilation time it is able to stop compilation raising appropriate errors, thereby inhibiting the successful creation of Java classes as long as the Java programmer does not correct erroneous code or supplies missing vital information. Java being a strictly typed language allows one to cast arguments to force the compiler to pick the method the Java programmer had in mind in the case that the compiler could choose one from a set of a signatures.

By contrast a bridge for an interpreted, dynamically typed language (like ooRexx) needs to reflect and infer at *runtime* in order to be able to choose the Java methods with the appropriate signature. In this particular case all primitive datatypes are (ooRexx/C) strings in the dynamically typed language such that at runtime each argument needs to be checked whether they contain valid values for conversion to primitive datatypes in the signature of a Java method candidate.

In the dynamically typed language there is no means available for casting that would be needed to help solve the presented problem (there is optionally a box()-routine in the bridge that allows the ooRexx programmer to explicitly determine which primitive type is to be used for picking an appropriate Java method).

The new code in the bridge (had to rewrite that part from scratch because of Java 9 and later) uses java.lang.reflect to pick an appropriate Constructor, Method or Field object. For Java 1.6 and 1.7 (although java.lang.invoke/MethodHandle got introduced with 1.7 I found cases where inaccessible types did not have accessible ancestors in the JRE) java.lang.reflect invocation will be carried out, for Java 1.8 and later java.lang.invoke/MethodHandle invocations gets employed.

Comparing the speed between invocations using java.lang.reflect and java.lang.invoke, java.lang.invoke wins, but by a quite small margin.

Upon further testunit tests the rewritten bridge using java.lang.invoke would execute all ooRexx programs like java.lang.reflect since Java 1.1 with one single exception so far which I reported here, where invocation via java.lang.reflect behaves differently to java.lang.invoke.

Because of this discrepancy the question would be: is this considered a bug that will be corrected? Would there be otherwise a solution possible in the bridge that could take care of situations like this, where the bridge has no context-information other than referring to Constructor, Method or Field objects with the supplied arguments?

A few coarse ideas to address this:

  * if the unreflect() method gets used then MH.invoke() should behave
    like in the java.lang.reflect case; this means that the same
    assumptions should govern MH.invoke(): reason being that if a
    java.lang.reflect object gets unreflected, also the established
    java.lang.reflect rules should keep working in this case for the
    MH.invoke* in order to remain fully compatible with each other,
      o if a MH gets created without unreflect() then this would not
        be necessary
  * the bridge could restrain itself to only use java.lang.reflect for
    invocations on Java 8 and higher instead of java.lang.invoke
      o clearly, the current Java development efforts in this corner
        are concentrated on java.lang.invoke, so it would be desirable
        to switch from java.lang.reflect to java.lang.invoke which is
        not possible as long as this problem (different behaviour in
        j.l.r vs. j.l.i) persists
  * do not fix this, but document this as an incompatibility with the
    ramification that existing programs in dynamically typed languages
    need to be rewritten (e.g. in this case creating an Array object
    from the List and supply that instead)
  * ... ?


P.S.: It would be great if Java implemented reflective invocations for dynamically typed languages as this would solve the problem for all such languages once and forever in a standard manner, rather than have every bridge implementor create his own implementation, which is challenging,  effortful and possibly error prone. In my case the assumptions that governed the original implementation for reflective Constructor, Method and Field access for over fifteen years got broken by Java 9, because j.l.r (notable 'setAccessible') changed in a fundamental aspect. If Java could be emplyoed instead, the Java supplied implementation would be able to adopt changes in the inner workings accordingly thereby insulating bridges from such details and problems.

On 08.06.2018 18:23, Peter Levart wrote:
I think what you found is a discrepancy between handling of varargs methods using reflection vs. method handle's .invokeWithArguments().

Reflection basically ignores the fact that some methods are varargs methods. It treats them exactly the same as if they had an array parameter (not a varargs parameter).

For example, Arrays.asList() is a varargs method:

    public static <T> List<T> asList(T ... elements)

But reflection treats is exactly the same as the following method:

    public static <T> List<T> asList(T[] elements)

That's because varargs were introduced to Java in Java 5 as a kind of compilation sugar, implemented as an array parameter, while reflection is an older beast and wasn't updated to behave differently with varargs methods.

MethodHandle(s) came later, in Java 7, and took into consideration varargs methods.

So where's the problem? When you say in Java:

        Method asListMethod = Arrays.class.getMethod("asList", Object[].class);
        String[] elements = {"a", "b", "c"};

            asListMethod.invoke(null, elements)

You get: IllegalArgumentException: wrong number of arguments

Because Method.invoke() is also a varargs method:

    public Object invoke(Object obj, Object... args)

... so java compiler (in order to be backwards source compatible with pre-varargs methods) treats this invocation in a way that just passes the String[] elements to the invoke method via the args parameter without any conversion (because String[] is a subtype of Object[]). Mathod.invoke then treats elements of the args[] array as individual verbatim parameters to be passed to the method when invoking it. As reflection treats Arrays.asList() as a normal no-varargs method which accepts a single Objet[] parameter and args[] contains 3 elements to be passed to a method with 3 parameters, exception occurs.

But when you use method handles and .invokeWithArguments():

        Method asListMethod = Arrays.class.getMethod("asList", Object[].class);         MethodHandle asListMH = MethodHandles.lookup().unreflect(asListMethod);

        String[] elements = {"a", "b", "c"};


... the following happens: MethodHandle.invokeWithArguments() is also a varargs method:

    public Object invokeWithArguments(Object... arguments)

... so java compiler (in order to be backwards source compatible with pre-varargs methods) treats this invocation in a way that just passes the String[] elements to the invokeWithArguments method via the single 'arguments' parameter without any conversion (because String[] is a subtype of Object[]). MethodHandle.invokeWithArguments therefore takes 3 array elements and tries to invoke the underlying asList method with them. It observes that Arrays.asList is a varargs method, so it wraps the 3 "wannabe parameters" with the Object[] and passes the array to the asList method as single parameter. The result of above code is therefore:

    [a, b, c]

If you want MethodHandle.invokeWithArguments() to treat "wannabe parameters" passed to it as verbatim parameters regardless of the variable/fixed arity of the method to be invoked, then you can transform the variable arity MethodHandle to a fixed arity MH:

        Method asListMethod = Arrays.class.getMethod("asList", Object[].class);

        String[] elements = {"a", "b", "c"};

            asListMethod.invoke(null, (Object) elements)

        MethodHandle asListMH = MethodHandles.lookup().unreflect(asListMethod);
        asListMH = asListMH.asFixedArity();

            asListMH.invokeWithArguments((Object) elements)

Here you have it. Both of the above invocations produce equal output:

    [a, b, c]
    [a, b, c]

What happens here is the following (will only describe the MH case - Method case is similar):

MethodHandle.invokeWithArguments() is a varargs method:

    public Object invokeWithArguments(Object... arguments)

... so java compiler this time wraps the '(Object) elements' value with an Object[] because the value is of Object type (which is not a subtype of Object[]) - no backwards compatibility needed here, varargs conversion kicks in in the javac. MethodHandle.invokeWithArguments therefore takes an array with 1 element (the element being 'elements' array) and tries to invoke the underlying asList method with it. This time it observes that the method is not a varargs method, (because MethodHandle was transformed to fixed arity method handle) so it passes the single 'elements' wannabee parameter to the Arrays.asList single Object[] parameter - this invocation works because fixed arity asList() takes Object[] and 'elements' is a String[]...

Hope this helps you understand what's going on.

Regards, Peter

On 06/08/18 12:07, Rony G. Flatscher wrote:
On 11.03.2018 20:22, Rony G. Flatscher wrote:
Well, still trying to find out what the reason is, that core reflection's invoke behaves differently to MethodHandle's invokeWithArguments in one single case so far (using the method java.utli.Arrays.asList(...)).

Here is a little Java program that excercises reflective access to "java.util.Arrays.asList​(T... a)" using core reflection, i.e. "Method.invoke​(Object obj, Object... args)" and "MethodHandle.invokeWithArguments​(Object... arguments)":

    import java.util.*;

    import java.lang.reflect.*;
    import java.lang.invoke.*;

    class DemoAsListProblem
         public static void main (String args[])
             String arrNames[]=new String[] { "anne", "bert", "celine"};
             System.out.println("[1] (main) arrNames=\""+arrNames+"\", 
             List listNames=Arrays.asList(arrNames);
             System.out.println("[2] (main) after invoking Arrays.asList(arrNames), 
             System.out.println("\ninvoking testReflective() ...\n");


         public static void testReflective()
             String arrNames[]=new String[] { "anne", "bert", "celine"};
             System.out.println("[3] (testReflective) arrNames=\""+arrNames+"\", 

             Method methAsList=null;
             List listNames=null;
             try {
                 Class paramTypes[]=new Class[] { (new Object[0]).getClass() };
                 System.out.println("--- (core reflection) Method object asList: 
                 System.out.println("\n--- (core reflection) now invoking 
Arrays.asList() via Method.invoke(...):");

                 listNames=(List) methAsList.invoke( null, (Object[]) new 
Object[]{arrNames} );   // static method
                 System.out.println("[4a] --- (CR) methAsList.invoke( null, (Object[]) new 
Object[]{arrNames} ) -> listNames=\""+listNames+"\"");

                 listNames=(List) methAsList.invoke( null, (Object)   new 
Object[]{arrNames} );   // static method
                 System.out.println("[4b] --- (CR) methAsList.invoke( null, (Object)   new 
Object[]{arrNames} ) -> listNames=\""+listNames+"\"");

    // "java.lang.IllegalArgumentException: wrong number of arguments":
    //            listNames=(List) methAsList.invoke( null, (Object[]) arrNames 
);   // static method
    //            System.out.println("[5a] --- (CR) methAsList.invoke( null, (Object[]) arrNames ) -> 

                 listNames=(List) methAsList.invoke( null, (Object)   arrNames 
);   // static method
                 System.out.println("[5b] --- (CR) methAsList.invoke( null, (Object)   arrNames )           
    -> listNames=\""+listNames+"\"");
             catch (Throwable t)
                 System.err.println("oops #1: "+t);

             System.out.println("\n--- (MH) now invoking Arrays.asList() via 
             MethodHandles.Lookup lookup=MethodHandles.lookup();
             MethodHandle mh=null;
             try {
                 System.out.println("--- (MH) unreflected MethodHandle mh: 

                 listNames=(List) mh.invokeWithArguments( (Object[]) new 
Object[]{arrNames} );
                 System.out.println("[6a] --- (MH) mh.invokeWithArguments(  (Object[]) new 
Object[]{arrNames} ) -> listNames=\""+listNames+"\"");

                 listNames=(List) mh.invokeWithArguments( (Object) new 
Object[]{arrNames} );
                 System.out.println("[6b] --- (MH) mh.invokeWithArguments(  (Object)   new 
Object[]{arrNames} ) -> listNames=\""+listNames+"\"");

                 listNames=(List) mh.invokeWithArguments( (Object[]) arrNames );
                 System.out.println("[7a] --- (MH) mh.invokeWithArguments(  (Object[]) arrNames )           
    -> listNames=\""+listNames+"\"");

                 listNames=(List) mh.invokeWithArguments( (Object) arrNames );
                 System.out.println("[7b] --- (MH) mh.invokeWithArguments(  (Object)   arrNames )           
    -> listNames=\""+listNames+"\"");

             catch (Throwable t)
                 System.err.println("oops #2: "+t);

Compiling and running it under 9.0.4 yields the following output:

    [1] (main) arrNames="[Ljava.lang.String;@27ddd392", 
    [2] (main) after invoking Arrays.asList(arrNames), listNames="[anne, bert, 

    invoking testReflective() ...

    [3] (testReflective) arrNames="[Ljava.lang.String;@2a18f23c", 
    --- (core reflection) Method object asList: public static java.util.List 

    --- (core reflection) now invoking Arrays.asList() via Method.invoke(...):
    [4a] --- (CR) methAsList.invoke( null, (Object[]) new Object[]{arrNames} ) -> 
listNames="[anne, bert, celine]"
    [4b] --- (CR) methAsList.invoke( null, (Object)   new Object[]{arrNames} ) -> 
    [5b] --- (CR) methAsList.invoke( null, (Object)   arrNames )               -> 
listNames="[anne, bert, celine]"

    --- (MH) now invoking Arrays.asList() via 
    --- (MH) unreflected MethodHandle mh: MethodHandle(Object[])List
    [6a] --- (MH) mh.invokeWithArguments(  (Object[]) new Object[]{arrNames} ) -> 
    [6b] --- (MH) mh.invokeWithArguments(  (Object)   new Object[]{arrNames} ) -> 
    [7a] --- (MH) mh.invokeWithArguments(  (Object[]) arrNames )               -> 
listNames="[anne, bert, celine]"
    [7b] --- (MH) mh.invokeWithArguments(  (Object)   arrNames )               -> 

So a String array is turned into a List using Arrays.asList(strArray). Doing it with core reflection yields different results to doing it with invokeWithArguments().

I would have expected that [4a] and [6a] would behave the same.

Using reflective invoke() and invokeWithArguments() has been working for my bridge for all the test units (using literally the same arguments in both cases) interchangeably, the one exception is Arrays.asList().

Maybe I am not seeing the obvious (having done too much "close-up tests" for quite some time now). So any hint, insight, help would be really appreciated!

As a few months have gone by without any follow-ups, I have been wondering whether there is a solution at all, if this is a problem rooted in the current implementation of MethodHandle methods not being 100% compatible with reflective invoke (which may imply that one needs to stick to use reflective invoke and forgo MethodHandle invokes for good).


mlvm-dev mailing list

Reply via email to