Re: About the native-lambda branch

[Daniel Sun]

Hi Jochen,

  Thanks a lot for your detailed explanation block by block!

> If you use the Closure code, then variables used in Closures are marked 
> as Reference even if they are only read 
  According to your suggestion, I've fixed the loading variable issue[1] 
:-)

  Thanks for your help again!

Cheers,
Daniel.Sun

[1]
https://github.com/apache/groovy/blob/8616d8967d847026102de999e50deaae1bdfc444/src/main/java/org/codehaus/groovy/classgen/asm/sc/StaticTypesLambdaWriter.java#L115-L131



--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html

Re: About the native-lambda branch

[Jochen Theodorou]

Am 15.01.2018 um 10:53 schrieb Daniel Sun:

Hi Jochen,

  `ArrayIndexOutOfBoundsException` is fixed. I encounter another
problem(i.e. How to load arguments according to some specified order): I
want to load local variables[1] according to the order in which the local
variables appear in lambda body. For example:
   
(1)

```
String x = 'x'
Integer y = 2
Stream.of(1, 2, 3).map(e -> '' + x + y + e) // Note the order of `x` and
`y`(`x` is before `y`)
```
I hope load `x`, then load `y` before invokedynamic.

(2)
```
String x = 'x'
Integer y = 2
Stream.of(1, 2, 3).map(e -> '' + y + x + e) // Note the order of `x` and
`y`(`y` is before `x`)
```
I hope  load `y`, then load `x` before invokedynamic.

 Here is how I try to archieve[2], but I can not get the expected
result[3], i.e. I get `groovy.lang.Reference` instances... not String
instances.


If you use the Closure code, then variables used in Closures are marked 
as Reference even if they are only read




 The following bytecode is generated for java code[4] by javac. The key
part is shown as follows and is what I want to generate via ASM utilities of
Groovy(e.g. `CompileStack`, `OperandStack`, etc):


what you do is for exmple


Parameter[] lambdaSharedVariableParameters = 
syntheticLambdaMethodNode.getNodeMetaData(LAMBDA_SHARED_VARIABLES);
for (int i = 0; i < lambdaSharedVariableParameters.length; i++) {
mv.visitVarInsn(ALOAD, i);

operandStack.doGroovyCast(lambdaSharedVariableParameters[i].getType().redirect());
//
operandStack.push(lambdaSharedVariableParameters[i].getType().redirect());
}


Assuming you manage to not wrap in a Reference, then the shared 
parameter might be a primitive. in that case aload is the wrong bytecode 
instruction. Please use CompileStack to handle the local variables instead.


[...]


The complete bytecode:
```
// class version 52.0 (52)
// access flags 0x21
public class Test2 {

[...]

   // access flags 0x9
   public static p()V
L0
 LINENUMBER 10 L0
 LDC "#"
 ASTORE 0


this creates a String and stores it in 0


L1
 LINENUMBER 12 L1
 ICONST_3
 ANEWARRAY java/lang/Integer
 DUP
 ICONST_0
 ICONST_1
 INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer;
 AASTORE
 DUP
 ICONST_1
 ICONST_2
 INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer;
 AASTORE
 DUP
 ICONST_2
 ICONST_3
 INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer;
 AASTORE


this creates an Integer[]{1,2,3} and leaves that on the operand stack


 INVOKESTATIC java/util/stream/Stream.of
([Ljava/lang/Object;)Ljava/util/stream/Stream;


this calls Stream.of with the Integer[], leaving a Stream object on the 
operand stack



 ALOAD 0


loads back the string from 0, now we have String on top and the Stream 
as second operand



 INVOKEDYNAMIC apply(Ljava/lang/String;)Ljava/util/function/Function; [
   // handle kind 0x6 : INVOKESTATIC >
java/lang/invoke/LambdaMetafactory.metafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite;
   // arguments:
   (Ljava/lang/Object;)Ljava/lang/Object;,
   // handle kind 0x6 : INVOKESTATIC
Test2.lambda$p$0(Ljava/lang/String;Ljava/lang/Integer;)Ljava/lang/String;,
   (Ljava/lang/Integer;)Ljava/lang/String;
 ]


hm I am not too familiar with the LambdaFactory logic so I cannot 
read this all too well. I see in the last two a parameters the constants 
for the lambda$p$0 and its minified MethodType. the lambda has a String 
and an Integer parameter and return a String. But because the String is 
bound through the factory to create the value for x, the resulting 
minified/cleanedup reference is (Ljava/lang/Integer;)Ljava/lang/String; 
instead.


As of why there is a (Ljava/lang/Object;)Ljava/lang/Object; I do not 
know. But the whole construct should take a String and give back a Function.




 INVOKEINTERFACE java/util/stream/Stream.map
(Ljava/util/function/Function;)Ljava/util/stream/Stream;


till here we had Stream, Function (the String was consumed by the 
lambdafactory call) on the stack, this calls map with the Function as 
argument on the Stream.



 POP


and here we forget about the return value


Does this description help you?

bye Jochen

Re: About the native-lambda branch

[Daniel Sun]

Hi Jochen,

  I still have a quesiton: How can I load variables via Groovy
utilities? Here[1] is my current way to load varaibles, it is not correct
though it can compile and run(i.e. the related test can not pass[2]):

   I tried the following way, but error occurred[3]...
```
int newRegister = 0;
Parameter[] lambdaSharedVariableParameters =
syntheticLambdaMethodNode.getNodeMetaData(LAMBDA_SHARED_VARIABLES);
for (Parameter parameter : lambdaSharedVariableParameters) {
ClassNode type = parameter.getType();
operandStack.load(parameter.getType(), newRegister);
// increment to next register, double/long are using two places
newRegister++;
if (type == ClassHelper.double_TYPE || type ==
ClassHelper.long_TYPE) newRegister++;
}
operandStack.remove(lambdaSharedVariableParameters.length);
```

Could you tell me how to load variables properly?

Cheers,
Daniel.Sun

[1]
https://github.com/apache/groovy/blob/fb8e3d10b9bcce46ebb474657d67036b14c2bffc/src/main/java/org/codehaus/groovy/classgen/asm/sc/StaticTypesLambdaWriter.java#L104-L109
[2]
https://github.com/apache/groovy/blob/fb8e3d10b9bcce46ebb474657d67036b14c2bffc/src/test/groovy/transform/stc/LambdaTest.groovy#L201-L222
[3]
```
java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
Test1.p()V @81: invokedynamic
  Reason:
Type 'groovy/lang/Reference' (current frame, stack[2]) is not assignable
to 'java/lang/String'
  Current Frame:
bci: @81
flags: { }
locals: { 'groovy/lang/Reference',
'org/codehaus/groovy/runtime/powerassert/ValueRecorder' }
stack: { 'java/util/List', 'java/util/stream/Stream',
'groovy/lang/Reference' }
  Bytecode:
0x000: 1222 bb00 245a 5fb7 0027 4b2a 57bb 0029
0x010: 59b7 002a 4c06 bd00 0459 0312 2c53 5904
0x020: 122e 5359 0512 3053 b800 3606 bd00 0459
0x030: 0304 b800 3c53 5904 05b8 003c 5359 0506
0x040: b800 3c53 b800 4259 2b5f 1243 b600 4757
0x050: 2aba 005d  b900 6102 0059 2b5f 1262
0x060: b600 4757 b800 6859 2b5f 1269 b600 4757
0x070: b900 6d02 0059 2b5f 126e b600 4757 b800
0x080: 7259 b800 772b 5f12 78b6 0047 5799 000a
0x090: 2bb6 007b a700 1512 7d2b b800 8301 b800
0x0a0: 87a7 0008 2bb6 007b bfb1   
  Exception Handler Table:
bci [21, 161] => handler: 164
  Stackmap Table:
append_frame(@151,Object[#36],Object[#41])
same_locals_1_stack_item_frame(@164,Object[#139])
same_frame(@169)

```



--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html

Re: About the native-lambda branch

[Jochen Theodorou]

Am 15.01.2018 um 15:21 schrieb Daniel Sun:

Hi Jesper,

  Thanks for your advice :-)


That's much easier than making the order dependent on the usage.

   Actually javac makes the order dependent on the local variables usage
in the lambda body. I'll follow its way for the time being because I can
reference how it make lambda work ;-)


yes, don't worry and use the standard mechanisms we have in the compiler 
here already, which means we do not reuse variable slots and order is 
dependent on usage. As long as the variable names and the line numbers 
are correct for the debugger later of course ;)


bye Jochen

Re: About the native-lambda branch

[Daniel Sun]

Hi Jesper,

 Thanks for your advice :-)

> That's much easier than making the order dependent on the usage.
  Actually javac makes the order dependent on the local variables usage
in the lambda body. I'll follow its way for the time being because I can
reference how it make lambda work ;-)

  As for some complex senarioes, I will try to support too later.
Currently, I only want to load variables properly via Groovy utilities to
get the expected result.

Cheers,
Daniel.Sun



--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html

Re: About the native-lambda branch

[Jesper Steen Møller]

Hi Daniel

> On 15 Jan 2018, at 10.53, Daniel Sun  wrote:
> 
> Hi Jochen,
> 
> `ArrayIndexOutOfBoundsException` is fixed. I encounter another
> problem(i.e. How to load arguments according to some specified order): I
> want to load local variables[1] according to the order in which the local
> variables appear in lambda body.

Don't do that :-)

Instead, you need to determine which method arguments and locals variables are 
needed by the lambda, and simply push all those in the declaration order. If 
they occur multiple times, you will need to filter anyway.

That's much easier than making the order dependent on the usage.

Also, you should really try to make some lower level tests, which tests the 
escape analysis directly, since you have so many different cases to look out 
for:
 - lambda use 'this' (explicitly or implicitly - controls whether the lambda 
implementation function should be static or not, and whether the this reference 
should be passed to the lambda)
 - lambda captures the enclosing method's parameters (passed as straight value 
if (effectively) final, or as a Reference if it changes)
 - lambda captures the enclosing method's local variables (ditto)
In case of lambdas nested in lambdas: Now you will have to address deal with 
inner lambdas capturing outer lambda's parameters and/or locals)

Shuffling/optimizing the order of the captured values is not worth it -- the 
JIT compiler will deal with it anyway.

Good luck!

-Jesper

Re: About the native-lambda branch

[Daniel Sun]

Hi Jochen,

 `ArrayIndexOutOfBoundsException` is fixed. I encounter another
problem(i.e. How to load arguments according to some specified order): I
want to load local variables[1] according to the order in which the local
variables appear in lambda body. For example:
  
(1)
```
String x = 'x'
Integer y = 2
Stream.of(1, 2, 3).map(e -> '' + x + y + e) // Note the order of `x` and
`y`(`x` is before `y`)
```
I hope load `x`, then load `y` before invokedynamic.

(2)
```
String x = 'x'
Integer y = 2
Stream.of(1, 2, 3).map(e -> '' + y + x + e) // Note the order of `x` and
`y`(`y` is before `x`)
```
I hope  load `y`, then load `x` before invokedynamic.

Here is how I try to archieve[2], but I can not get the expected
result[3], i.e. I get `groovy.lang.Reference` instances... not String
instances.

The following bytecode is generated for java code[4] by javac. The key
part is shown as follows and is what I want to generate via ASM utilities of
Groovy(e.g. `CompileStack`, `OperandStack`, etc):

```
ALOAD 0
INVOKEDYNAMIC apply(Ljava/lang/String;)Ljava/util/function/Function; [
  // handle kind 0x6 : INVOKESTATIC
 
java/lang/invoke/LambdaMetafactory.metafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite;
  // arguments:
  (Ljava/lang/Object;)Ljava/lang/Object;, 
  // handle kind 0x6 : INVOKESTATIC
 
Test2.lambda$p$0(Ljava/lang/String;Ljava/lang/Integer;)Ljava/lang/String;, 
  (Ljava/lang/Integer;)Ljava/lang/String;
]
```

The complete bytecode:
```
// class version 52.0 (52)
// access flags 0x21
public class Test2 {

  // compiled from: Test2.java
  // access flags 0x19
  public final static INNERCLASS java/lang/invoke/MethodHandles$Lookup
java/lang/invoke/MethodHandles Lookup

  // access flags 0x1
  public ()V
   L0
LINENUMBER 4 L0
ALOAD 0
INVOKESPECIAL java/lang/Object. ()V
RETURN
MAXSTACK = 1
MAXLOCALS = 1

  // access flags 0x9
  public static main([Ljava/lang/String;)V
   L0
LINENUMBER 6 L0
INVOKESTATIC Test2.p ()V
   L1
LINENUMBER 7 L1
RETURN
MAXSTACK = 0
MAXLOCALS = 1

  // access flags 0x9
  public static p()V
   L0
LINENUMBER 10 L0
LDC "#"
ASTORE 0
   L1
LINENUMBER 12 L1
ICONST_3
ANEWARRAY java/lang/Integer
DUP
ICONST_0
ICONST_1
INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer;
AASTORE
DUP
ICONST_1
ICONST_2
INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer;
AASTORE
DUP
ICONST_2
ICONST_3
INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer;
AASTORE
INVOKESTATIC java/util/stream/Stream.of
([Ljava/lang/Object;)Ljava/util/stream/Stream;
ALOAD 0
INVOKEDYNAMIC apply(Ljava/lang/String;)Ljava/util/function/Function; [
  // handle kind 0x6 : INVOKESTATIC
 
java/lang/invoke/LambdaMetafactory.metafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite;
  // arguments:
  (Ljava/lang/Object;)Ljava/lang/Object;, 
  // handle kind 0x6 : INVOKESTATIC
 
Test2.lambda$p$0(Ljava/lang/String;Ljava/lang/Integer;)Ljava/lang/String;, 
  (Ljava/lang/Integer;)Ljava/lang/String;
]
INVOKEINTERFACE java/util/stream/Stream.map
(Ljava/util/function/Function;)Ljava/util/stream/Stream;
POP
   L2
LINENUMBER 13 L2
RETURN
MAXSTACK = 4
MAXLOCALS = 1

  // access flags 0x100A
  private static synthetic
lambda$p$0(Ljava/lang/String;Ljava/lang/Integer;)Ljava/lang/String;
   L0
LINENUMBER 12 L0
NEW java/lang/StringBuilder
DUP
INVOKESPECIAL java/lang/StringBuilder. ()V
ALOAD 0
INVOKEVIRTUAL java/lang/StringBuilder.append
(Ljava/lang/String;)Ljava/lang/StringBuilder;
ALOAD 1
INVOKEVIRTUAL java/lang/StringBuilder.append
(Ljava/lang/Object;)Ljava/lang/StringBuilder;
INVOKEVIRTUAL java/lang/StringBuilder.toString ()Ljava/lang/String;
ARETURN
MAXSTACK = 2
MAXLOCALS = 2
}

```

Thanks in advance for your help :-)

Cheers,
Daniel.Sun

[1]
https://github.com/apache/groovy/blob/fb8e3d10b9bcce46ebb474657d67036b14c2bffc/src/test/groovy/transform/stc/LambdaTest.groovy#L218
[2]
https://github.com/apache/groovy/blob/fb8e3d10b9bcce46ebb474657d67036b14c2bffc/src/main/java/org/codehaus/groovy/classgen/asm/sc/StaticTypesLambdaWriter.java#L104-L109
[3]
https://github.com/apache/groovy/blob/fb8e3d10b9bcce46ebb474657d67036b14c2bffc/src/test/groovy/transform/stc/LambdaTest.groovy#L203
[4] 
```
import java.util.stream.Collectors;
import java.util.stream.Stream;

public class Test2 {
public static void main(String[] args) {
p();
}

public static void p() {
String x = "#";

Stream.of(1, 

Re: About the native-lambda branch

[Daniel.Sun]

Hi Jochen,

  Thanks for your detailed explanation, which is very helpful to me.

Cheers,
Daniel.Sun




--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html

Re: About the native-lambda branch

[Jochen Theodorou]

On 14.01.2018 01:10, Daniel Sun wrote:

Hi Jochen,


What I wish for in a static compile lambda is the following:
* bar is a parameter to the method generated for the lambda


I am trying to make native lambda support sharing local variables, but ASM
reports the following error[1]:
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
at org.objectweb.asm.Frame.merge(Frame.java:1501)

Have you ever encountered similar problems? Looking forward to your
suggestions :-)


most likely two frames with a differing number of operand stack 
elements. Like


aload 0
dup
ifnonnull goto Y
X:
dup
y:


first we load this, then we double it. the conditional jump will remove 
one. So once we arrive in y we would have one element on the stack, if 
the condition does not apply we go to X, then we have two elements on 
the operand stack and once we arrive in Y we still do.


bye jochen

Re: About the native-lambda branch

[Daniel.Sun]

It seems that I found where go wrong.
The shared local variables in generated method body should have been
replaced with parameters. Let me try later :-)

Cheers,
Daniel.Sun




--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html

Re: About the native-lambda branch

[Daniel Sun]

Hi Jochen,

> What I wish for in a static compile lambda is the following: 
> * bar is a parameter to the method generated for the lambda

I am trying to make native lambda support sharing local variables, but ASM
reports the following error[1]:
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
at org.objectweb.asm.Frame.merge(Frame.java:1501)

Have you ever encountered similar problems? Looking forward to your
suggestions :-)

Cheers,
Daniel.Sun

[1]
https://github.com/apache/groovy/blob/b6ea72dbf2ee4ab63fb1b96569de609679807a34/src/test/groovy/transform/stc/LambdaTest.groovy#L258



--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html

Re: About the native-lambda branch

[Jochen Theodorou]

On 13.01.2018 04:07, Nathan Harvey wrote:

Sure thing. Here's a Java example:

void sample(Function fn) {
System.out.println("fn");
}

void sample(Supplier sp) {
System.out.println("sp");
}

These methods can exist side by side, and are called correctly even in cases
of Lambda, eg:

sample(s -> 123); // fn
sample(() -> 123); // sp

On the other hand, take this Groovy code:


​def sample(Function fn) {
 println "fn"
}

def sample(Supplier sp) {
 println "sp"
}


in theory we can resolve this, because we do know how many parameters 
the Closure has and in sample({ s -> 1 }) we know we have one, thus it 
should be the Function. What we cannot do with Closure is the following:


interface I1 {
  String m(String s)
}
interace I2 {
  Integer m(Integer i)
}
def c(I1 i1){}
def c(I2 i2){}
c(i -> i+1)
c(i -> i.toString())

What should not work even in Java is the following:

interface I1 {
  String m(String s)
}
interace I2 {
  String m(Integer i)
}
def c(I1 i1){}
def c(I2 i2){}
c(i -> (i+1).toString())
c(i -> i.toString())

that is because here the return type will give no additional 
information, so only the parameter type can work and i.toString() works 
for both, i+1 should in theory also work for both.


bye blackdrag

Re: About the native-lambda branch

[Daniel Sun]

Hi Jochen,

 Here is an example:

```
import java.util.stream.Collectors;
import java.util.stream.Stream;

public class Test1 {
public static void main(String[] args) {
p();
}

public static void p() {
 Stream.of(1, 2, 3).map(e -> e + 1).collect(Collectors.toList());
}
}
```

Its bytecode is shown as follows, note `  public final static INNERCLASS
java/lang/invoke/MethodHandles$Lookup java/lang/invoke/MethodHandles
Lookup`, which I found is useless, so I removed it and all code works still
well.

```
// class version 52.0 (52)
// access flags 0x21
public class Test1 {

  // compiled from: Test1.java
  // access flags 0x19
  public final static INNERCLASS java/lang/invoke/MethodHandles$Lookup
java/lang/invoke/MethodHandles Lookup

  // access flags 0x1
  public ()V
   L0
LINENUMBER 4 L0
ALOAD 0
INVOKESPECIAL java/lang/Object. ()V
RETURN
MAXSTACK = 1
MAXLOCALS = 1

  // access flags 0x9
  public static main([Ljava/lang/String;)V
   L0
LINENUMBER 6 L0
INVOKESTATIC Test1.p ()V
   L1
LINENUMBER 7 L1
RETURN
MAXSTACK = 0
MAXLOCALS = 1

  // access flags 0x9
  public static p()V
   L0
LINENUMBER 10 L0
ICONST_3
ANEWARRAY java/lang/Integer
DUP
ICONST_0
ICONST_1
INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer;
AASTORE
DUP
ICONST_1
ICONST_2
INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer;
AASTORE
DUP
ICONST_2
ICONST_3
INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer;
AASTORE
INVOKESTATIC java/util/stream/Stream.of
([Ljava/lang/Object;)Ljava/util/stream/Stream;
INVOKEDYNAMIC apply()Ljava/util/function/Function; [
  // handle kind 0x6 : INVOKESTATIC
 
java/lang/invoke/LambdaMetafactory.metafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite;
  // arguments:
  (Ljava/lang/Object;)Ljava/lang/Object;,
  // handle kind 0x6 : INVOKESTATIC
  Test1.lambda$p$0(Ljava/lang/Integer;)Ljava/lang/Integer;,
  (Ljava/lang/Integer;)Ljava/lang/Integer;
]
INVOKEINTERFACE java/util/stream/Stream.map
(Ljava/util/function/Function;)Ljava/util/stream/Stream;
INVOKESTATIC java/util/stream/Collectors.toList
()Ljava/util/stream/Collector;
INVOKEINTERFACE java/util/stream/Stream.collect
(Ljava/util/stream/Collector;)Ljava/lang/Object;
POP
   L1
LINENUMBER 11 L1
RETURN
MAXSTACK = 4
MAXLOCALS = 0

  // access flags 0x100A
  private static synthetic
lambda$p$0(Ljava/lang/Integer;)Ljava/lang/Integer;
   L0
LINENUMBER 10 L0
ALOAD 0
INVOKEVIRTUAL java/lang/Integer.intValue ()I
ICONST_1
IADD
INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer;
ARETURN
MAXSTACK = 2
MAXLOCALS = 1
}

```

Cheers,
Daniel.Sun



--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html

Re: About the native-lambda branch

[Nathan Harvey]

Sure thing. Here's a Java example:

void sample(Function fn) {
System.out.println("fn");
}

void sample(Supplier sp) {
System.out.println("sp");
}

These methods can exist side by side, and are called correctly even in cases
of Lambda, eg:

sample(s -> 123); // fn
sample(() -> 123); // sp

On the other hand, take this Groovy code:


​def sample(Function fn) {
println "fn"
}

def sample(Supplier sp) {
println "sp"
}


With attempted invocation:

sample({ s -> 1 })​

This produces an error: 

Cannot resolve which method to invoke for [class Script1$_run_closure1] due
to overlapping prototypes between:
[interface java.util.function.Function]
[interface java.util.function.Supplier]



--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html

Re: About the native-lambda branch

[Nathan Harvey]

One of the largest reasons I found myself preferring Groovy over Java is the
quasi-final scoping restrictions of Lambdas. Please do not let this behavior
exist in Groovy. It makes functional programming very hard.

On the other hand, lambdas are superior for functional programming because
they can be differentiated with Java 8's functional type classes quite
easily. In a perfect world, this would be possible with Closures as well.



--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html

Re: About the native-lambda branch

[Jochen Theodorou]

Am 12.01.2018 um 18:54 schrieb Russel Winder:

On Fri, 2018-01-12 at 05:54 -0700, Daniel.Sun wrote:



[…]

 As you see, the latest implementation is to generate method for lambda
at the class generation stage, in addition new inner classes will be
generated for each lambda.

[…]


I haven't read the code, I am reacting to the above paragraph a bit out of
context. Nonetheless…

The whole point of Java lambdas and use of invokedynamic is to avoid any
classes at all, so the mention of using an inner class worries me. The
anonymous inner class route was tried and rejected in favour of runtime code
generation triggered via an ignoredynamic – I do not know the full details,
hopefully I am not wrong.


ignoredynamic... lol... It is especially funny for me because the 
invokedynamic logic does allow you to be more dynamic in your method 
selection and callsite logic in general, but it first and foremost uses 
static information for this. You have to apply a small trick to get a 
real dynamic call out of this. Means I find ignoredynamic quite fitting 
actually.



The point here is for me, Groovy should use the same strategy as Java for what
is a Java feature.


It does not have to be 100% the same as Java. Not the same limitations. 
But we should be careful if it means bad things for the performance, it 
is a prize we have to decide about if we want to pay it.


bye Jochen

Re: About the native-lambda branch

[Jochen Theodorou]

Am 12.01.2018 um 15:00 schrieb Daniel Sun:

Hi Jochen,


but I think you should then not use the closure inner class mechanism.

I remain the implementation for non-native lambda[1] to make lambda work in
legacy code, where closures are widely used. For example, `[1, 2,
3].collect(e -> e + 1)`


ok, then forget about my other mail from a minute ago ;)

We have to be a bit careful here imho. Defining an inner class, means 
additional class loading times, accessing private fields becomes also 
more tricky. If you want to be able to be "equal" to the javac code as 
much as possible, then you should not base this on closure so much. In 
fact I think our closures should be based on lambdas instead. If we 
already had solved all the lambda problems, then I would change Closure 
accordingly actually. Yes, for a first step basing them on Closure is 
nice, but that can easily backfire.


For example

class SomeClass {
  private  foo;
  def m(bar) {
SomeFunctionalInterface x =
   (it -> println "I am a lambda: $foo and $bar")
return x
  }
}

Following the Closure route of dynamic Groovy means here the following:
(1) I create a Closure for the lambda as inner class, one such class per 
lambda

(2) I will ignore the overhead of the Closure instance generation
(3) I use the asType mechanism to create a dynamic proxy for my Closure
(4) If somebody creates a sublcass of Someclass accessing the private 
field foo gets complicated

(5) actually using the lambda would go through doCall/call
(6) accessing bar is realized through

and for the static case
* 1-3 and 5 will still apply. For point (4) there will be a bridging 
method (synthetic public static) in SomeClass, which is directly called 
to access the field. the static compiler is adding this method on its 
own (very very very late)


What I wish for in a static compile lambda is the following:
* bar is a parameter to the method generated for the lambda
* there is no groovy dynamic call involved to get to the lambda code
* the proxy for SomeFunctionalInterface is generated by the lambda factory
* foo is accessed directly through get field

And what I would wish for for the future Closure is. There is no inner 
class Closure anymore, instead Closure is just a holder for symbolic 
references and maybe some instances. Bar for example would still come in 
through the constructor. The Closure instance itself would be a 
parameter to the lambda method implementation (together with owner and 
such). And finally: there is a way to define a stateless Closure (but 
maybe the lambdas are the way for this)



I thought you need that only to enable reflection to find your inner
classes.

Could you make it a bit detail? To be honest, I can not get your words...


If you use reflection on a class you can ask this class what inner 
classes it has. For this there is in bytecode an entry in the inner 
class table. The inner class has an outer class attribute which can be 
used by Reflection to query for the enclosing class (similar for 
enclosing method). For the JVM itself I think those all are not 
required. The JVM does not care if a class is Bar or Foo$Bar and if the 
constructor takes an instance of Foo or not unlesss you call that 
constructor. The JVM even requires those bridge methods to access the 
private fields of the enclosing class. Which is also the reason why 
Groovy had this information wrong for a long time and not very many did 
really care.


bye Jochen

Re: About the native-lambda branch

[Russel Winder]

On Fri, 2018-01-12 at 05:54 -0700, Daniel.Sun wrote:
> 
[…]
> As you see, the latest implementation is to generate method for lambda
> at the class generation stage, in addition new inner classes will be
> generated for each lambda.
> 
> […]

I haven't read the code, I am reacting to the above paragraph a bit out of
context. Nonetheless…

The whole point of Java lambdas and use of invokedynamic is to avoid any
classes at all, so the mention of using an inner class worries me. The
anonymous inner class route was tried and rejected in favour of runtime code
generation triggered via an ignoredynamic – I do not know the full details,
hopefully I am not wrong.

The point here is for me, Groovy should use the same strategy as Java for what
is a Java feature.

-- 
Russel.
==
Dr Russel Winder  t: +44 20 7585 2200
41 Buckmaster Roadm: +44 7770 465 077
London SW11 1EN, UK   w: www.russel.org.uk


signature.asc
Description: This is a digitally signed message part

Re: About the native-lambda branch

[Jochen Theodorou]

Am 12.01.2018 um 14:13 schrieb Daniel.Sun:

Hi Jesper,

   The bytecode generation for lambda of the current implementation
should be somehow same with javac does, because I reference the ASM code,
which is decomplied from bytecode generated by javac.

   If you have some spare time to contribute, glad to work with you again
;-)


can you give an example where javac creates an inner class at compile 
time for the lambda? Because I know no such case


bye Jochen

Re: About the native-lambda branch

[Daniel Sun]

Hi Jochen,

>  I did also not understand the innclass attribute visit...
I've cleaned up the useless code:
https://github.com/apache/groovy/commit/6aeaa1c5a7863af5d2c126a5bdf9da9aff2a8db6

Cheers,
Daniel.Sun



--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html

Re: About the native-lambda branch

[Jesper Steen Møller]

Hi Jochen and Daniel

> On 12 Jan 2018, at 12.48, Jochen Theodorou  wrote:
> 
> 
> 
> Am 12.01.2018 um 04:05 schrieb Daniel Sun:
>> Hi Nathan,
>>> What's will the differences be between closures and lambdas?
>> The native lambda will have better performance than closure. In addition,
>> native lambda, which conforms to Java specification,  is less versatile than
>> closure, which is really powerful...
> 
> but I think you should then not use the closure inner class mechanism. I had 
> a bit a look at your code, but there is a ton of things missing. I did also 
> not understand the innclass attribute visit... I thought you need that only 
> to enable reflection to find your inner classes. MethodHandles.Lookup is no 
> such thing. And then I would also question the early transformation you do 
> where you store the lambda as body in a method node... which resides in the 
> current class and which may have a name conflicting with an existing name... 
> and a name check is something I would not do in that ast builder. In fact I 
> would do this logic just before or even in ACG. Otherwise you will always 
> have trouble with code that transforms the lambda expression. And then again 
> we miss totally the handling of local variables, not in the lambda. And 
> actually there is a performance related decision to do here. if we follow the 
> closure logic, there will be a wrapper containing the value of the local 
> variable, which is then used in the lambda. This costs. Not much, but it 
> does. Since in Java you have only quasi final variables allowed, you do not 
> need any wrapper. Not using the wrapper means less Closure implementation 
> code we can use. But this wrapper also has quite special handling in many 
> areas, like the VariableScopeVisitor. Oh yes... should you go with one inner 
> class for all lambdas, then you have to add bridge methods to be able to 
> access private fields in the outer class.
> 

I don't think we need to generate bridge classes at all. I'm not sure if you 
are familiar with how lambdas are implemented by javac, but they also uses a 
method for each lambda expression, and capture the rest at indy time. Javac 
alsot differentiates the lambda implementatoin function based on what is 
captured: The lambda method is an instance method if it captures anything 
nonstatic, otherwise it can be a static method (and thus possibly recycled).

For capturing locals, it would be optimal to do the same "effectively final" 
logic similar to what Java does, so we could simply capture the value itself.
If we couldn't guarantee that (which ranges from tricky to impossible), then 
we'd fallback to using groovy.lang.Reference like closures do.

Am I making sense? Otherwise, it'd be better to talk "in code", I guess.

-Jesper

Re: About the native-lambda branch

[Jochen Theodorou]

Am 12.01.2018 um 04:05 schrieb Daniel Sun:

Hi Nathan,


What's will the differences be between closures and lambdas?

The native lambda will have better performance than closure. In addition,
native lambda, which conforms to Java specification,  is less versatile than
closure, which is really powerful...


but I think you should then not use the closure inner class mechanism. I 
had a bit a look at your code, but there is a ton of things missing. I 
did also not understand the innclass attribute visit... I thought you 
need that only to enable reflection to find your inner classes. 
MethodHandles.Lookup is no such thing. And then I would also question 
the early transformation you do where you store the lambda as body in a 
method node... which resides in the current class and which may have a 
name conflicting with an existing name... and a name check is something 
I would not do in that ast builder. In fact I would do this logic just 
before or even in ACG. Otherwise you will always have trouble with code 
that transforms the lambda expression. And then again we miss totally 
the handling of local variables, not in the lambda. And actually there 
is a performance related decision to do here. if we follow the closure 
logic, there will be a wrapper containing the value of the local 
variable, which is then used in the lambda. This costs. Not much, but it 
does. Since in Java you have only quasi final variables allowed, you do 
not need any wrapper. Not using the wrapper means less Closure 
implementation code we can use. But this wrapper also has quite special 
handling in many areas, like the VariableScopeVisitor. Oh yes... should 
you go with one inner class for all lambdas, then you have to add bridge 
methods to be able to access private fields in the outer class.


bye Jochen

Re: About the native-lambda branch

[Daniel Sun]

Hi Nathan,

> What's will the differences be between closures and lambdas? 
The native lambda will have better performance than closure. In addition,
native lambda, which conforms to Java specification,  is less versatile than
closure, which is really powerful...

Cheers,
Daniel.Sun



--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html

Re: About the native-lambda branch

[Daniel Sun]

Time is the problem, but I'll try to set aside some time to push the progress
;-)

Cheers,
Daniel.Sun



--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html

Re: About the native-lambda branch

[Nathan Harvey]

Great work, Daniel. What's will the differences be between closures and
lambdas? I love that you can have fine control over closure scoping, but
lambdas play very well with functional interfaces, which are very important!



--
Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html

Re: About the native-lambda branch

[Graeme Rocher]

Awesome to see progress on this. Keep it up!

On Thu, Jan 11, 2018 at 9:07 AM, Daniel Sun  wrote:
> Hi all,
>
>   I created the native-lambda branch and pushed the first commit to
> support very basic native lambda in the static compilation mode[1]. After
> native lambda is fully supported in the static compilation mode, I will try
> to make native lambda work in the dynamic compilation mode(I wish I would
> have enough spare time to complete...)
>
>   Currently some synthetic methods are generated while parsing, so they
> are lack of type information. it should be deferred. In addition, I am
> thinking about generating another new class with the synthetic methods for
> each lambda expression.
>
>   Any help to improve the native lambda is welcome and appreciated!
>
> Cheers,
> Daniel.Sun
>
> [1] https://github.com/apache/groovy/tree/native-lambda
>
>
>
> --
> Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html



-- 
Graeme Rocher