Re: About the native-lambda branch
Hi Jochen, Thanks a lot for your detailed explanation block by block! > If you use the Closure code, then variables used in Closures are marked > as Reference even if they are only read According to your suggestion, I've fixed the loading variable issue[1] :-) Thanks for your help again! Cheers, Daniel.Sun [1] https://github.com/apache/groovy/blob/8616d8967d847026102de999e50deaae1bdfc444/src/main/java/org/codehaus/groovy/classgen/asm/sc/StaticTypesLambdaWriter.java#L115-L131 -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: About the native-lambda branch
Am 15.01.2018 um 10:53 schrieb Daniel Sun: Hi Jochen, `ArrayIndexOutOfBoundsException` is fixed. I encounter another problem(i.e. How to load arguments according to some specified order): I want to load local variables[1] according to the order in which the local variables appear in lambda body. For example: (1) ``` String x = 'x' Integer y = 2 Stream.of(1, 2, 3).map(e -> '' + x + y + e) // Note the order of `x` and `y`(`x` is before `y`) ``` I hope load `x`, then load `y` before invokedynamic. (2) ``` String x = 'x' Integer y = 2 Stream.of(1, 2, 3).map(e -> '' + y + x + e) // Note the order of `x` and `y`(`y` is before `x`) ``` I hope load `y`, then load `x` before invokedynamic. Here is how I try to archieve[2], but I can not get the expected result[3], i.e. I get `groovy.lang.Reference` instances... not String instances. If you use the Closure code, then variables used in Closures are marked as Reference even if they are only read The following bytecode is generated for java code[4] by javac. The key part is shown as follows and is what I want to generate via ASM utilities of Groovy(e.g. `CompileStack`, `OperandStack`, etc): what you do is for exmple Parameter[] lambdaSharedVariableParameters = syntheticLambdaMethodNode.getNodeMetaData(LAMBDA_SHARED_VARIABLES); for (int i = 0; i < lambdaSharedVariableParameters.length; i++) { mv.visitVarInsn(ALOAD, i); operandStack.doGroovyCast(lambdaSharedVariableParameters[i].getType().redirect()); // operandStack.push(lambdaSharedVariableParameters[i].getType().redirect()); } Assuming you manage to not wrap in a Reference, then the shared parameter might be a primitive. in that case aload is the wrong bytecode instruction. Please use CompileStack to handle the local variables instead. [...] The complete bytecode: ``` // class version 52.0 (52) // access flags 0x21 public class Test2 { [...] // access flags 0x9 public static p()V L0 LINENUMBER 10 L0 LDC "#" ASTORE 0 this creates a String and stores it in 0 L1 LINENUMBER 12 L1 ICONST_3 ANEWARRAY java/lang/Integer DUP ICONST_0 ICONST_1 INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer; AASTORE DUP ICONST_1 ICONST_2 INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer; AASTORE DUP ICONST_2 ICONST_3 INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer; AASTORE this creates an Integer[]{1,2,3} and leaves that on the operand stack INVOKESTATIC java/util/stream/Stream.of ([Ljava/lang/Object;)Ljava/util/stream/Stream; this calls Stream.of with the Integer[], leaving a Stream object on the operand stack ALOAD 0 loads back the string from 0, now we have String on top and the Stream as second operand INVOKEDYNAMIC apply(Ljava/lang/String;)Ljava/util/function/Function; [ // handle kind 0x6 : INVOKESTATIC > java/lang/invoke/LambdaMetafactory.metafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite; // arguments: (Ljava/lang/Object;)Ljava/lang/Object;, // handle kind 0x6 : INVOKESTATIC Test2.lambda$p$0(Ljava/lang/String;Ljava/lang/Integer;)Ljava/lang/String;, (Ljava/lang/Integer;)Ljava/lang/String; ] hm I am not too familiar with the LambdaFactory logic so I cannot read this all too well. I see in the last two a parameters the constants for the lambda$p$0 and its minified MethodType. the lambda has a String and an Integer parameter and return a String. But because the String is bound through the factory to create the value for x, the resulting minified/cleanedup reference is (Ljava/lang/Integer;)Ljava/lang/String; instead. As of why there is a (Ljava/lang/Object;)Ljava/lang/Object; I do not know. But the whole construct should take a String and give back a Function. INVOKEINTERFACE java/util/stream/Stream.map (Ljava/util/function/Function;)Ljava/util/stream/Stream; till here we had Stream, Function (the String was consumed by the lambdafactory call) on the stack, this calls map with the Function as argument on the Stream. POP and here we forget about the return value Does this description help you? bye Jochen
Re: About the native-lambda branch
Hi Jochen, I still have a quesiton: How can I load variables via Groovy utilities? Here[1] is my current way to load varaibles, it is not correct though it can compile and run(i.e. the related test can not pass[2]): I tried the following way, but error occurred[3]... ``` int newRegister = 0; Parameter[] lambdaSharedVariableParameters = syntheticLambdaMethodNode.getNodeMetaData(LAMBDA_SHARED_VARIABLES); for (Parameter parameter : lambdaSharedVariableParameters) { ClassNode type = parameter.getType(); operandStack.load(parameter.getType(), newRegister); // increment to next register, double/long are using two places newRegister++; if (type == ClassHelper.double_TYPE || type == ClassHelper.long_TYPE) newRegister++; } operandStack.remove(lambdaSharedVariableParameters.length); ``` Could you tell me how to load variables properly? Cheers, Daniel.Sun [1] https://github.com/apache/groovy/blob/fb8e3d10b9bcce46ebb474657d67036b14c2bffc/src/main/java/org/codehaus/groovy/classgen/asm/sc/StaticTypesLambdaWriter.java#L104-L109 [2] https://github.com/apache/groovy/blob/fb8e3d10b9bcce46ebb474657d67036b14c2bffc/src/test/groovy/transform/stc/LambdaTest.groovy#L201-L222 [3] ``` java.lang.VerifyError: Bad type on operand stack Exception Details: Location: Test1.p()V @81: invokedynamic Reason: Type 'groovy/lang/Reference' (current frame, stack[2]) is not assignable to 'java/lang/String' Current Frame: bci: @81 flags: { } locals: { 'groovy/lang/Reference', 'org/codehaus/groovy/runtime/powerassert/ValueRecorder' } stack: { 'java/util/List', 'java/util/stream/Stream', 'groovy/lang/Reference' } Bytecode: 0x000: 1222 bb00 245a 5fb7 0027 4b2a 57bb 0029 0x010: 59b7 002a 4c06 bd00 0459 0312 2c53 5904 0x020: 122e 5359 0512 3053 b800 3606 bd00 0459 0x030: 0304 b800 3c53 5904 05b8 003c 5359 0506 0x040: b800 3c53 b800 4259 2b5f 1243 b600 4757 0x050: 2aba 005d b900 6102 0059 2b5f 1262 0x060: b600 4757 b800 6859 2b5f 1269 b600 4757 0x070: b900 6d02 0059 2b5f 126e b600 4757 b800 0x080: 7259 b800 772b 5f12 78b6 0047 5799 000a 0x090: 2bb6 007b a700 1512 7d2b b800 8301 b800 0x0a0: 87a7 0008 2bb6 007b bfb1 Exception Handler Table: bci [21, 161] => handler: 164 Stackmap Table: append_frame(@151,Object[#36],Object[#41]) same_locals_1_stack_item_frame(@164,Object[#139]) same_frame(@169) ``` -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: About the native-lambda branch
Am 15.01.2018 um 15:21 schrieb Daniel Sun: Hi Jesper, Thanks for your advice :-) That's much easier than making the order dependent on the usage. Actually javac makes the order dependent on the local variables usage in the lambda body. I'll follow its way for the time being because I can reference how it make lambda work ;-) yes, don't worry and use the standard mechanisms we have in the compiler here already, which means we do not reuse variable slots and order is dependent on usage. As long as the variable names and the line numbers are correct for the debugger later of course ;) bye Jochen
Re: About the native-lambda branch
Hi Jesper, Thanks for your advice :-) > That's much easier than making the order dependent on the usage. Actually javac makes the order dependent on the local variables usage in the lambda body. I'll follow its way for the time being because I can reference how it make lambda work ;-) As for some complex senarioes, I will try to support too later. Currently, I only want to load variables properly via Groovy utilities to get the expected result. Cheers, Daniel.Sun -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: About the native-lambda branch
Hi Daniel > On 15 Jan 2018, at 10.53, Daniel Sunwrote: > > Hi Jochen, > > `ArrayIndexOutOfBoundsException` is fixed. I encounter another > problem(i.e. How to load arguments according to some specified order): I > want to load local variables[1] according to the order in which the local > variables appear in lambda body. Don't do that :-) Instead, you need to determine which method arguments and locals variables are needed by the lambda, and simply push all those in the declaration order. If they occur multiple times, you will need to filter anyway. That's much easier than making the order dependent on the usage. Also, you should really try to make some lower level tests, which tests the escape analysis directly, since you have so many different cases to look out for: - lambda use 'this' (explicitly or implicitly - controls whether the lambda implementation function should be static or not, and whether the this reference should be passed to the lambda) - lambda captures the enclosing method's parameters (passed as straight value if (effectively) final, or as a Reference if it changes) - lambda captures the enclosing method's local variables (ditto) In case of lambdas nested in lambdas: Now you will have to address deal with inner lambdas capturing outer lambda's parameters and/or locals) Shuffling/optimizing the order of the captured values is not worth it -- the JIT compiler will deal with it anyway. Good luck! -Jesper
Re: About the native-lambda branch
Hi Jochen, `ArrayIndexOutOfBoundsException` is fixed. I encounter another problem(i.e. How to load arguments according to some specified order): I want to load local variables[1] according to the order in which the local variables appear in lambda body. For example: (1) ``` String x = 'x' Integer y = 2 Stream.of(1, 2, 3).map(e -> '' + x + y + e) // Note the order of `x` and `y`(`x` is before `y`) ``` I hope load `x`, then load `y` before invokedynamic. (2) ``` String x = 'x' Integer y = 2 Stream.of(1, 2, 3).map(e -> '' + y + x + e) // Note the order of `x` and `y`(`y` is before `x`) ``` I hope load `y`, then load `x` before invokedynamic. Here is how I try to archieve[2], but I can not get the expected result[3], i.e. I get `groovy.lang.Reference` instances... not String instances. The following bytecode is generated for java code[4] by javac. The key part is shown as follows and is what I want to generate via ASM utilities of Groovy(e.g. `CompileStack`, `OperandStack`, etc): ``` ALOAD 0 INVOKEDYNAMIC apply(Ljava/lang/String;)Ljava/util/function/Function; [ // handle kind 0x6 : INVOKESTATIC java/lang/invoke/LambdaMetafactory.metafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite; // arguments: (Ljava/lang/Object;)Ljava/lang/Object;, // handle kind 0x6 : INVOKESTATIC Test2.lambda$p$0(Ljava/lang/String;Ljava/lang/Integer;)Ljava/lang/String;, (Ljava/lang/Integer;)Ljava/lang/String; ] ``` The complete bytecode: ``` // class version 52.0 (52) // access flags 0x21 public class Test2 { // compiled from: Test2.java // access flags 0x19 public final static INNERCLASS java/lang/invoke/MethodHandles$Lookup java/lang/invoke/MethodHandles Lookup // access flags 0x1 public ()V L0 LINENUMBER 4 L0 ALOAD 0 INVOKESPECIAL java/lang/Object. ()V RETURN MAXSTACK = 1 MAXLOCALS = 1 // access flags 0x9 public static main([Ljava/lang/String;)V L0 LINENUMBER 6 L0 INVOKESTATIC Test2.p ()V L1 LINENUMBER 7 L1 RETURN MAXSTACK = 0 MAXLOCALS = 1 // access flags 0x9 public static p()V L0 LINENUMBER 10 L0 LDC "#" ASTORE 0 L1 LINENUMBER 12 L1 ICONST_3 ANEWARRAY java/lang/Integer DUP ICONST_0 ICONST_1 INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer; AASTORE DUP ICONST_1 ICONST_2 INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer; AASTORE DUP ICONST_2 ICONST_3 INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer; AASTORE INVOKESTATIC java/util/stream/Stream.of ([Ljava/lang/Object;)Ljava/util/stream/Stream; ALOAD 0 INVOKEDYNAMIC apply(Ljava/lang/String;)Ljava/util/function/Function; [ // handle kind 0x6 : INVOKESTATIC java/lang/invoke/LambdaMetafactory.metafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite; // arguments: (Ljava/lang/Object;)Ljava/lang/Object;, // handle kind 0x6 : INVOKESTATIC Test2.lambda$p$0(Ljava/lang/String;Ljava/lang/Integer;)Ljava/lang/String;, (Ljava/lang/Integer;)Ljava/lang/String; ] INVOKEINTERFACE java/util/stream/Stream.map (Ljava/util/function/Function;)Ljava/util/stream/Stream; POP L2 LINENUMBER 13 L2 RETURN MAXSTACK = 4 MAXLOCALS = 1 // access flags 0x100A private static synthetic lambda$p$0(Ljava/lang/String;Ljava/lang/Integer;)Ljava/lang/String; L0 LINENUMBER 12 L0 NEW java/lang/StringBuilder DUP INVOKESPECIAL java/lang/StringBuilder. ()V ALOAD 0 INVOKEVIRTUAL java/lang/StringBuilder.append (Ljava/lang/String;)Ljava/lang/StringBuilder; ALOAD 1 INVOKEVIRTUAL java/lang/StringBuilder.append (Ljava/lang/Object;)Ljava/lang/StringBuilder; INVOKEVIRTUAL java/lang/StringBuilder.toString ()Ljava/lang/String; ARETURN MAXSTACK = 2 MAXLOCALS = 2 } ``` Thanks in advance for your help :-) Cheers, Daniel.Sun [1] https://github.com/apache/groovy/blob/fb8e3d10b9bcce46ebb474657d67036b14c2bffc/src/test/groovy/transform/stc/LambdaTest.groovy#L218 [2] https://github.com/apache/groovy/blob/fb8e3d10b9bcce46ebb474657d67036b14c2bffc/src/main/java/org/codehaus/groovy/classgen/asm/sc/StaticTypesLambdaWriter.java#L104-L109 [3] https://github.com/apache/groovy/blob/fb8e3d10b9bcce46ebb474657d67036b14c2bffc/src/test/groovy/transform/stc/LambdaTest.groovy#L203 [4] ``` import java.util.stream.Collectors; import java.util.stream.Stream; public class Test2 { public static void main(String[] args) { p(); } public static void p() { String x = "#"; Stream.of(1,
Re: About the native-lambda branch
Hi Jochen, Thanks for your detailed explanation, which is very helpful to me. Cheers, Daniel.Sun -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: About the native-lambda branch
On 14.01.2018 01:10, Daniel Sun wrote: Hi Jochen, What I wish for in a static compile lambda is the following: * bar is a parameter to the method generated for the lambda I am trying to make native lambda support sharing local variables, but ASM reports the following error[1]: Caused by: java.lang.ArrayIndexOutOfBoundsException: -1 at org.objectweb.asm.Frame.merge(Frame.java:1501) Have you ever encountered similar problems? Looking forward to your suggestions :-) most likely two frames with a differing number of operand stack elements. Like aload 0 dup ifnonnull goto Y X: dup y: first we load this, then we double it. the conditional jump will remove one. So once we arrive in y we would have one element on the stack, if the condition does not apply we go to X, then we have two elements on the operand stack and once we arrive in Y we still do. bye jochen
Re: About the native-lambda branch
It seems that I found where go wrong. The shared local variables in generated method body should have been replaced with parameters. Let me try later :-) Cheers, Daniel.Sun -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: About the native-lambda branch
Hi Jochen, > What I wish for in a static compile lambda is the following: > * bar is a parameter to the method generated for the lambda I am trying to make native lambda support sharing local variables, but ASM reports the following error[1]: Caused by: java.lang.ArrayIndexOutOfBoundsException: -1 at org.objectweb.asm.Frame.merge(Frame.java:1501) Have you ever encountered similar problems? Looking forward to your suggestions :-) Cheers, Daniel.Sun [1] https://github.com/apache/groovy/blob/b6ea72dbf2ee4ab63fb1b96569de609679807a34/src/test/groovy/transform/stc/LambdaTest.groovy#L258 -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: About the native-lambda branch
On 13.01.2018 04:07, Nathan Harvey wrote: Sure thing. Here's a Java example: void sample(Functionfn) { System.out.println("fn"); } void sample(Supplier sp) { System.out.println("sp"); } These methods can exist side by side, and are called correctly even in cases of Lambda, eg: sample(s -> 123); // fn sample(() -> 123); // sp On the other hand, take this Groovy code: def sample(Function fn) { println "fn" } def sample(Supplier sp) { println "sp" } in theory we can resolve this, because we do know how many parameters the Closure has and in sample({ s -> 1 }) we know we have one, thus it should be the Function. What we cannot do with Closure is the following: interface I1 { String m(String s) } interace I2 { Integer m(Integer i) } def c(I1 i1){} def c(I2 i2){} c(i -> i+1) c(i -> i.toString()) What should not work even in Java is the following: interface I1 { String m(String s) } interace I2 { String m(Integer i) } def c(I1 i1){} def c(I2 i2){} c(i -> (i+1).toString()) c(i -> i.toString()) that is because here the return type will give no additional information, so only the parameter type can work and i.toString() works for both, i+1 should in theory also work for both. bye blackdrag
Re: About the native-lambda branch
Hi Jochen, Here is an example: ``` import java.util.stream.Collectors; import java.util.stream.Stream; public class Test1 { public static void main(String[] args) { p(); } public static void p() { Stream.of(1, 2, 3).map(e -> e + 1).collect(Collectors.toList()); } } ``` Its bytecode is shown as follows, note ` public final static INNERCLASS java/lang/invoke/MethodHandles$Lookup java/lang/invoke/MethodHandles Lookup`, which I found is useless, so I removed it and all code works still well. ``` // class version 52.0 (52) // access flags 0x21 public class Test1 { // compiled from: Test1.java // access flags 0x19 public final static INNERCLASS java/lang/invoke/MethodHandles$Lookup java/lang/invoke/MethodHandles Lookup // access flags 0x1 public ()V L0 LINENUMBER 4 L0 ALOAD 0 INVOKESPECIAL java/lang/Object. ()V RETURN MAXSTACK = 1 MAXLOCALS = 1 // access flags 0x9 public static main([Ljava/lang/String;)V L0 LINENUMBER 6 L0 INVOKESTATIC Test1.p ()V L1 LINENUMBER 7 L1 RETURN MAXSTACK = 0 MAXLOCALS = 1 // access flags 0x9 public static p()V L0 LINENUMBER 10 L0 ICONST_3 ANEWARRAY java/lang/Integer DUP ICONST_0 ICONST_1 INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer; AASTORE DUP ICONST_1 ICONST_2 INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer; AASTORE DUP ICONST_2 ICONST_3 INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer; AASTORE INVOKESTATIC java/util/stream/Stream.of ([Ljava/lang/Object;)Ljava/util/stream/Stream; INVOKEDYNAMIC apply()Ljava/util/function/Function; [ // handle kind 0x6 : INVOKESTATIC java/lang/invoke/LambdaMetafactory.metafactory(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite; // arguments: (Ljava/lang/Object;)Ljava/lang/Object;, // handle kind 0x6 : INVOKESTATIC Test1.lambda$p$0(Ljava/lang/Integer;)Ljava/lang/Integer;, (Ljava/lang/Integer;)Ljava/lang/Integer; ] INVOKEINTERFACE java/util/stream/Stream.map (Ljava/util/function/Function;)Ljava/util/stream/Stream; INVOKESTATIC java/util/stream/Collectors.toList ()Ljava/util/stream/Collector; INVOKEINTERFACE java/util/stream/Stream.collect (Ljava/util/stream/Collector;)Ljava/lang/Object; POP L1 LINENUMBER 11 L1 RETURN MAXSTACK = 4 MAXLOCALS = 0 // access flags 0x100A private static synthetic lambda$p$0(Ljava/lang/Integer;)Ljava/lang/Integer; L0 LINENUMBER 10 L0 ALOAD 0 INVOKEVIRTUAL java/lang/Integer.intValue ()I ICONST_1 IADD INVOKESTATIC java/lang/Integer.valueOf (I)Ljava/lang/Integer; ARETURN MAXSTACK = 2 MAXLOCALS = 1 } ``` Cheers, Daniel.Sun -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: About the native-lambda branch
Sure thing. Here's a Java example: void sample(Functionfn) { System.out.println("fn"); } void sample(Supplier sp) { System.out.println("sp"); } These methods can exist side by side, and are called correctly even in cases of Lambda, eg: sample(s -> 123); // fn sample(() -> 123); // sp On the other hand, take this Groovy code: def sample(Function fn) { println "fn" } def sample(Supplier sp) { println "sp" } With attempted invocation: sample({ s -> 1 }) This produces an error: Cannot resolve which method to invoke for [class Script1$_run_closure1] due to overlapping prototypes between: [interface java.util.function.Function] [interface java.util.function.Supplier] -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: About the native-lambda branch
One of the largest reasons I found myself preferring Groovy over Java is the quasi-final scoping restrictions of Lambdas. Please do not let this behavior exist in Groovy. It makes functional programming very hard. On the other hand, lambdas are superior for functional programming because they can be differentiated with Java 8's functional type classes quite easily. In a perfect world, this would be possible with Closures as well. -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: About the native-lambda branch
Am 12.01.2018 um 18:54 schrieb Russel Winder: On Fri, 2018-01-12 at 05:54 -0700, Daniel.Sun wrote: […] As you see, the latest implementation is to generate method for lambda at the class generation stage, in addition new inner classes will be generated for each lambda. […] I haven't read the code, I am reacting to the above paragraph a bit out of context. Nonetheless… The whole point of Java lambdas and use of invokedynamic is to avoid any classes at all, so the mention of using an inner class worries me. The anonymous inner class route was tried and rejected in favour of runtime code generation triggered via an ignoredynamic – I do not know the full details, hopefully I am not wrong. ignoredynamic... lol... It is especially funny for me because the invokedynamic logic does allow you to be more dynamic in your method selection and callsite logic in general, but it first and foremost uses static information for this. You have to apply a small trick to get a real dynamic call out of this. Means I find ignoredynamic quite fitting actually. The point here is for me, Groovy should use the same strategy as Java for what is a Java feature. It does not have to be 100% the same as Java. Not the same limitations. But we should be careful if it means bad things for the performance, it is a prize we have to decide about if we want to pay it. bye Jochen
Re: About the native-lambda branch
Am 12.01.2018 um 15:00 schrieb Daniel Sun: Hi Jochen, but I think you should then not use the closure inner class mechanism. I remain the implementation for non-native lambda[1] to make lambda work in legacy code, where closures are widely used. For example, `[1, 2, 3].collect(e -> e + 1)` ok, then forget about my other mail from a minute ago ;) We have to be a bit careful here imho. Defining an inner class, means additional class loading times, accessing private fields becomes also more tricky. If you want to be able to be "equal" to the javac code as much as possible, then you should not base this on closure so much. In fact I think our closures should be based on lambdas instead. If we already had solved all the lambda problems, then I would change Closure accordingly actually. Yes, for a first step basing them on Closure is nice, but that can easily backfire. For example class SomeClass { private foo; def m(bar) { SomeFunctionalInterface x = (it -> println "I am a lambda: $foo and $bar") return x } } Following the Closure route of dynamic Groovy means here the following: (1) I create a Closure for the lambda as inner class, one such class per lambda (2) I will ignore the overhead of the Closure instance generation (3) I use the asType mechanism to create a dynamic proxy for my Closure (4) If somebody creates a sublcass of Someclass accessing the private field foo gets complicated (5) actually using the lambda would go through doCall/call (6) accessing bar is realized through and for the static case * 1-3 and 5 will still apply. For point (4) there will be a bridging method (synthetic public static) in SomeClass, which is directly called to access the field. the static compiler is adding this method on its own (very very very late) What I wish for in a static compile lambda is the following: * bar is a parameter to the method generated for the lambda * there is no groovy dynamic call involved to get to the lambda code * the proxy for SomeFunctionalInterface is generated by the lambda factory * foo is accessed directly through get field And what I would wish for for the future Closure is. There is no inner class Closure anymore, instead Closure is just a holder for symbolic references and maybe some instances. Bar for example would still come in through the constructor. The Closure instance itself would be a parameter to the lambda method implementation (together with owner and such). And finally: there is a way to define a stateless Closure (but maybe the lambdas are the way for this) I thought you need that only to enable reflection to find your inner classes. Could you make it a bit detail? To be honest, I can not get your words... If you use reflection on a class you can ask this class what inner classes it has. For this there is in bytecode an entry in the inner class table. The inner class has an outer class attribute which can be used by Reflection to query for the enclosing class (similar for enclosing method). For the JVM itself I think those all are not required. The JVM does not care if a class is Bar or Foo$Bar and if the constructor takes an instance of Foo or not unlesss you call that constructor. The JVM even requires those bridge methods to access the private fields of the enclosing class. Which is also the reason why Groovy had this information wrong for a long time and not very many did really care. bye Jochen
Re: About the native-lambda branch
On Fri, 2018-01-12 at 05:54 -0700, Daniel.Sun wrote: > […] > As you see, the latest implementation is to generate method for lambda > at the class generation stage, in addition new inner classes will be > generated for each lambda. > > […] I haven't read the code, I am reacting to the above paragraph a bit out of context. Nonetheless… The whole point of Java lambdas and use of invokedynamic is to avoid any classes at all, so the mention of using an inner class worries me. The anonymous inner class route was tried and rejected in favour of runtime code generation triggered via an ignoredynamic – I do not know the full details, hopefully I am not wrong. The point here is for me, Groovy should use the same strategy as Java for what is a Java feature. -- Russel. == Dr Russel Winder t: +44 20 7585 2200 41 Buckmaster Roadm: +44 7770 465 077 London SW11 1EN, UK w: www.russel.org.uk signature.asc Description: This is a digitally signed message part
Re: About the native-lambda branch
Am 12.01.2018 um 14:13 schrieb Daniel.Sun: Hi Jesper, The bytecode generation for lambda of the current implementation should be somehow same with javac does, because I reference the ASM code, which is decomplied from bytecode generated by javac. If you have some spare time to contribute, glad to work with you again ;-) can you give an example where javac creates an inner class at compile time for the lambda? Because I know no such case bye Jochen
Re: About the native-lambda branch
Hi Jochen, > I did also not understand the innclass attribute visit... I've cleaned up the useless code: https://github.com/apache/groovy/commit/6aeaa1c5a7863af5d2c126a5bdf9da9aff2a8db6 Cheers, Daniel.Sun -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: About the native-lambda branch
Hi Jochen and Daniel > On 12 Jan 2018, at 12.48, Jochen Theodorouwrote: > > > > Am 12.01.2018 um 04:05 schrieb Daniel Sun: >> Hi Nathan, >>> What's will the differences be between closures and lambdas? >> The native lambda will have better performance than closure. In addition, >> native lambda, which conforms to Java specification, is less versatile than >> closure, which is really powerful... > > but I think you should then not use the closure inner class mechanism. I had > a bit a look at your code, but there is a ton of things missing. I did also > not understand the innclass attribute visit... I thought you need that only > to enable reflection to find your inner classes. MethodHandles.Lookup is no > such thing. And then I would also question the early transformation you do > where you store the lambda as body in a method node... which resides in the > current class and which may have a name conflicting with an existing name... > and a name check is something I would not do in that ast builder. In fact I > would do this logic just before or even in ACG. Otherwise you will always > have trouble with code that transforms the lambda expression. And then again > we miss totally the handling of local variables, not in the lambda. And > actually there is a performance related decision to do here. if we follow the > closure logic, there will be a wrapper containing the value of the local > variable, which is then used in the lambda. This costs. Not much, but it > does. Since in Java you have only quasi final variables allowed, you do not > need any wrapper. Not using the wrapper means less Closure implementation > code we can use. But this wrapper also has quite special handling in many > areas, like the VariableScopeVisitor. Oh yes... should you go with one inner > class for all lambdas, then you have to add bridge methods to be able to > access private fields in the outer class. > I don't think we need to generate bridge classes at all. I'm not sure if you are familiar with how lambdas are implemented by javac, but they also uses a method for each lambda expression, and capture the rest at indy time. Javac alsot differentiates the lambda implementatoin function based on what is captured: The lambda method is an instance method if it captures anything nonstatic, otherwise it can be a static method (and thus possibly recycled). For capturing locals, it would be optimal to do the same "effectively final" logic similar to what Java does, so we could simply capture the value itself. If we couldn't guarantee that (which ranges from tricky to impossible), then we'd fallback to using groovy.lang.Reference like closures do. Am I making sense? Otherwise, it'd be better to talk "in code", I guess. -Jesper
Re: About the native-lambda branch
Am 12.01.2018 um 04:05 schrieb Daniel Sun: Hi Nathan, What's will the differences be between closures and lambdas? The native lambda will have better performance than closure. In addition, native lambda, which conforms to Java specification, is less versatile than closure, which is really powerful... but I think you should then not use the closure inner class mechanism. I had a bit a look at your code, but there is a ton of things missing. I did also not understand the innclass attribute visit... I thought you need that only to enable reflection to find your inner classes. MethodHandles.Lookup is no such thing. And then I would also question the early transformation you do where you store the lambda as body in a method node... which resides in the current class and which may have a name conflicting with an existing name... and a name check is something I would not do in that ast builder. In fact I would do this logic just before or even in ACG. Otherwise you will always have trouble with code that transforms the lambda expression. And then again we miss totally the handling of local variables, not in the lambda. And actually there is a performance related decision to do here. if we follow the closure logic, there will be a wrapper containing the value of the local variable, which is then used in the lambda. This costs. Not much, but it does. Since in Java you have only quasi final variables allowed, you do not need any wrapper. Not using the wrapper means less Closure implementation code we can use. But this wrapper also has quite special handling in many areas, like the VariableScopeVisitor. Oh yes... should you go with one inner class for all lambdas, then you have to add bridge methods to be able to access private fields in the outer class. bye Jochen
Re: About the native-lambda branch
Hi Nathan, > What's will the differences be between closures and lambdas? The native lambda will have better performance than closure. In addition, native lambda, which conforms to Java specification, is less versatile than closure, which is really powerful... Cheers, Daniel.Sun -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: About the native-lambda branch
Time is the problem, but I'll try to set aside some time to push the progress ;-) Cheers, Daniel.Sun -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: About the native-lambda branch
Great work, Daniel. What's will the differences be between closures and lambdas? I love that you can have fine control over closure scoping, but lambdas play very well with functional interfaces, which are very important! -- Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html
Re: About the native-lambda branch
Awesome to see progress on this. Keep it up! On Thu, Jan 11, 2018 at 9:07 AM, Daniel Sunwrote: > Hi all, > > I created the native-lambda branch and pushed the first commit to > support very basic native lambda in the static compilation mode[1]. After > native lambda is fully supported in the static compilation mode, I will try > to make native lambda work in the dynamic compilation mode(I wish I would > have enough spare time to complete...) > > Currently some synthetic methods are generated while parsing, so they > are lack of type information. it should be deferred. In addition, I am > thinking about generating another new class with the synthetic methods for > each lambda expression. > > Any help to improve the native lambda is welcome and appreciated! > > Cheers, > Daniel.Sun > > [1] https://github.com/apache/groovy/tree/native-lambda > > > > -- > Sent from: http://groovy.329449.n5.nabble.com/Groovy-Dev-f372993.html -- Graeme Rocher