Re: What can we improve in JSR292 for Java 9?
Hi John, comment inlined Le 26 août 2015 00:30:00 CEST, John Rose john.r.r...@oracle.com a écrit : On Feb 25, 2015, at 6:29 PM, John Rose john.r.r...@oracle.com wrote: Maybe this is general enough: MHs.loop(init, predicate, body)(*a) = { let i = init(*a); while (predicate(i, *a)) { i = body(i, *a); } return i; } ...where the type of i depends on init, and if init returns void then you have a classic side-effect-only loop. FTR, I've been thinking more about this one, with help from Michael Haupt. The above loop is pretty good, except for a couple of things: 'i' ought to be tuple of loop-varying values, and the exit path needs a more general post-processing step, a finalizer that corresponds to the initializers that set up the loop variables. A generalized (linear) loop has two kinds of values flowing within it: loop-invariant and loop-varying. (The control predicate has a special role, but may be classed as loop-varying. The finalizer is also special, and loop-invariant.) Loop-varying values can be inputs, control values (array indices, etc.), and outputs, but can all, I think, be modeled as step functions using a common signature. This takes us to something like the C for-loop, but with a sprinkle of tuples, with a finishing bow around it to tie it up: for ((v1,v2,v3,…) = inits; predicate(v1…); (v1…) = step(v1…)) { … }; return finish(v1…); Loop-local but invariant values can be modeled as non-updated loop variables, as in this C idiom: for (int i, limit = computeLimit(); i limit; i++) … Pure side effects can be modeled as void-valued step functions, so we want to allow void as a corner case for loop variable types. (I.e., where there is a step but not a value.) I don't want to make any more special provision for pure side effects (such as a unique Runnable-like body for a loop), both because it isn't necessary, and because it encourages sloppy side-effect-driven loops. If we allow each loop variable to have its own predicate and finalizer, we can properly model the effects of ordered guards and special early-exit paths. loop({init*}, {pred*}, {step*}, {fini*}) := lambda (arg*) { let var* = init*(arg*) for (;;) { for ((v,p,s,f) in (var*, pred*, step*, fini*)) if (!p(var*, arg*)) return f(var*, arg*) v = s(var*, arg*) // all vars help update each var } } } All invocations are exact. As a bonus we can easily derive post-checked loops (do-while), and time-shifted index variables (p++) simply by adding more variables and/or steps. but i think i prefer to have pre and post updates instead of several predicates/finishers. Like this: loop({init*}, {prestep*}, {pred}, {poststep*}, {fini}) := lambda (arg*) { let var* = init*(arg*) for (;;) { foreach v in vars: v = prestep(var*, args*) if (!pred(var*, arg*)) return fini(var*, arg*) foreach v in vars: v = poststep(var*, args*) } it's less powerful but it seems you can encode all your examples of loops too. Constraints on parameter and return types: - There must be the same number of inits, preds, steps, and finis. - There must be at least one input function (e.g., a pred, so that the loop has a chance to exit). - Parameter lists of the init functions must all be the same. - Parameter lists of the pred, step, and fini functions must all be the same. - Parameter lists of the step functions must be a suffix of the parameter lists of the init functions. - Parameters of the step functions must be the same as the return types of the init functions (with voids removed), followed by the common suffix. - Return types of the predicates must all be boolean. - Return types of the finalizers must all be the same (and can be void). - The return type of each init function must match the return type of the corresponding step function (a var type or void). The result, a looping method handle, has a return type taken from the finalizer function(s), and argument types taken from the init functions. The loop has one iteration variable for each non-void init/step function pair. Each step/pred/fini call takes all of the iteration variable values, followed by all the external (loop-invariant) arguments. No throws are caught by the loop; throws cause the loop to terminate abnormally. In some loops, such as find first match, the predicate naturally follows the step function, because it needs to evaluate the result of the step. To model post-checks (and the classic do/while shape), a second void-valued loop variable is required: loop(...) := lambda (arg*) { let (var, _) = (null, void) for (;;) { if (_) return _ // trivial predicate: exit branch never taken var = s(_, arg*) // pick up next search key if (!p(var, arg*)) // did we find it? return f(var, arg*) // extract value from search key as needed _ = _(var, arg*) // void value, no side effect } } This variable-doubling could be compressed out if there were a further boolean (per pred/step
Re: What can we improve in JSR292 for Java 9?
On Aug 26, 2015, at 6:06 AM, Rémi Forax fo...@univ-mlv.fr wrote: it's less powerful but it seems you can encode all your examples of loops too. No, I don't think so. Multiple pred/fini pairs implement multiple loop exits. Any loop with an if-return in it has an exit distinct from the loop's main test. And each exit requires a different fini function, to encode a special result. The match loop has this multi-exit character: for (int i; i a.len; i++) { Object e = a[i]; if (ok(e)) return e; // first fini function; could also be e.payload } return null; // second fini function; could also be throwNotFound or Optional.none Sometimes even type predicate logic expands into multi-exit code, if the loop needs to return a value that encodes which sub-predicate caused the exit. — John ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
yes, you're right, so if think i prefer to 'allow a free intermixing of any number of steps (var updates) with any other number of pred/fini pairs' which requires less brain power to understand that the double encoding you are proposing :) Something like: public interface InnerLoopBuilder { // variableIndex in [0..inits.length[ void addVariableUpdate(int variableIndex, MethodHandle step); void addLoopExit(MethodHandle predicate, MethodHandle finisher); } MethodHandle loopMH(MethodHandle[] inits, ConsumerInnerLoopBuilder consumer) { ... } cheers, Rémi - Mail original - De: John Rose john.r.r...@oracle.com À: Rémi Forax fo...@univ-mlv.fr Cc: Da Vinci Machine Project mlvm-dev@openjdk.java.net Envoyé: Jeudi 27 Août 2015 01:28:20 Objet: Re: What can we improve in JSR292 for Java 9? On Aug 26, 2015, at 6:06 AM, Rémi Forax fo...@univ-mlv.fr wrote: it's less powerful but it seems you can encode all your examples of loops too. No, I don't think so. Multiple pred/fini pairs implement multiple loop exits. Any loop with an if-return in it has an exit distinct from the loop's main test. And each exit requires a different fini function, to encode a special result. The match loop has this multi-exit character: for (int i; i a.len; i++) { Object e = a[i]; if (ok(e)) return e; // first fini function; could also be e.payload } return null; // second fini function; could also be throwNotFound or Optional.none Sometimes even type predicate logic expands into multi-exit code, if the loop needs to return a value that encodes which sub-predicate caused the exit. — John ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
On Feb 25, 2015, at 6:29 PM, John Rose john.r.r...@oracle.com wrote: Maybe this is general enough: MHs.loop(init, predicate, body)(*a) = { let i = init(*a); while (predicate(i, *a)) { i = body(i, *a); } return i; } ...where the type of i depends on init, and if init returns void then you have a classic side-effect-only loop. FTR, I've been thinking more about this one, with help from Michael Haupt. The above loop is pretty good, except for a couple of things: 'i' ought to be tuple of loop-varying values, and the exit path needs a more general post-processing step, a finalizer that corresponds to the initializers that set up the loop variables. A generalized (linear) loop has two kinds of values flowing within it: loop-invariant and loop-varying. (The control predicate has a special role, but may be classed as loop-varying. The finalizer is also special, and loop-invariant.) Loop-varying values can be inputs, control values (array indices, etc.), and outputs, but can all, I think, be modeled as step functions using a common signature. This takes us to something like the C for-loop, but with a sprinkle of tuples, with a finishing bow around it to tie it up: for ((v1,v2,v3,…) = inits; predicate(v1…); (v1…) = step(v1…)) { … }; return finish(v1…); Loop-local but invariant values can be modeled as non-updated loop variables, as in this C idiom: for (int i, limit = computeLimit(); i limit; i++) … Pure side effects can be modeled as void-valued step functions, so we want to allow void as a corner case for loop variable types. (I.e., where there is a step but not a value.) I don't want to make any more special provision for pure side effects (such as a unique Runnable-like body for a loop), both because it isn't necessary, and because it encourages sloppy side-effect-driven loops. If we allow each loop variable to have its own predicate and finalizer, we can properly model the effects of ordered guards and special early-exit paths. loop({init*}, {pred*}, {step*}, {fini*}) := lambda (arg*) { let var* = init*(arg*) for (;;) { for ((v,p,s,f) in (var*, pred*, step*, fini*)) if (!p(var*, arg*)) return f(var*, arg*) v = s(var*, arg*) // all vars help update each var } } } All invocations are exact. As a bonus we can easily derive post-checked loops (do-while), and time-shifted index variables (p++) simply by adding more variables and/or steps. Constraints on parameter and return types: - There must be the same number of inits, preds, steps, and finis. - There must be at least one input function (e.g., a pred, so that the loop has a chance to exit). - Parameter lists of the init functions must all be the same. - Parameter lists of the pred, step, and fini functions must all be the same. - Parameter lists of the step functions must be a suffix of the parameter lists of the init functions. - Parameters of the step functions must be the same as the return types of the init functions (with voids removed), followed by the common suffix. - Return types of the predicates must all be boolean. - Return types of the finalizers must all be the same (and can be void). - The return type of each init function must match the return type of the corresponding step function (a var type or void). The result, a looping method handle, has a return type taken from the finalizer function(s), and argument types taken from the init functions. The loop has one iteration variable for each non-void init/step function pair. Each step/pred/fini call takes all of the iteration variable values, followed by all the external (loop-invariant) arguments. No throws are caught by the loop; throws cause the loop to terminate abnormally. In some loops, such as find first match, the predicate naturally follows the step function, because it needs to evaluate the result of the step. To model post-checks (and the classic do/while shape), a second void-valued loop variable is required: loop(...) := lambda (arg*) { let (var, _) = (null, void) for (;;) { if (_) return _ // trivial predicate: exit branch never taken var = s(_, arg*) // pick up next search key if (!p(var, arg*)) // did we find it? return f(var, arg*) // extract value from search key as needed _ = _(var, arg*) // void value, no side effect } } This variable-doubling could be compressed out if there were a further boolean (per pred/step pair) which says which order to run them in. To me this seems like too much ad hoc structure; such structure tends to creep over time as we discover other important patterns. I think it would be better to support optionality for common no op loop components, as follows: Optionalities: - If an init is null, a constant zero (or null/false/void) function is supplied by inspecting the return type of the corresponding step function. - If a pred is null, a constant true function is supplied. - If a step is null, a suitable
Re: What can we improve in JSR292 for Java 9?
Am 08.03.2015 18:39, schrieb Peter Levart: On 03/08/2015 01:38 PM, Jochen Theodorou wrote: Am 08.03.2015 12:16, schrieb Remi Forax: [...] You need to restrict the set of method handles that are allowed to be installed inside a CallSite corresponding to such kind of invokedynamic. Something like: you can do any transformations you want on the arguments but not on the receiver (no drop, no permute, no filter) and then you need to call unconditionally a method handle created using findSpecial on the super class. unconditionally, like no guard? @Jochen: There can be GWT, but it can only select one of target/fallback that are both from the restricted set of MHs. [...] So Jochen, can you point me to an algorithm used for MultiMethods dispatch in Groovy and I'll try to prepare a proof-of-concept that uses it? You can use the Java dispatch rules, only that you take runtime types for receiver and arguments, instead of the static ones. I suggest you also ignore vargs (life is complicated enough). Groovy does some more conversions and other fancy stuff, but I think the above is already complicated enough, especially because you have to solve the problem of a fallback that has to cause a new method selection. If you can solve that problem, then I see use for super-init-calls with that style. So far we have to install a dispatcher handle through the static callsite bootstrapper, because of the lack of runtimetype information. Which then will do the actual method selection of course. That means the first handle we install is not the target and the first invocation of the real target method is done from inside my runtime in the dynamic bootstrapper, since I cannot simply fallback to the static case anymore. It is the same for any case in which the guards for checking the runtime types fail. A PIC should have similar problems. bye Jochen -- Jochen blackdrag Theodorou - Groovy Project Tech Lead blog: http://blackdragsview.blogspot.com/ german groovy discussion newsgroup: de.comp.lang.misc For Groovy programming sources visit http://groovy-lang.org ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
On 02/26/2015 01:02 AM, Charles Oliver Nutter wrote: After talking with folks at the Jfokus VM Summit, it seems like there's a number of nice-to-have and a few need-to-have features we'd like to see get into java.lang.invoke. Vladimir suggested I start a thread on these features. A few from me: * A loop handle :-) Given a body and a test, run the body until the test is false. I'm guessing there's a good reason we don't have this already. * try/finally as a core atom of MethodHandles API. Libraries like invokebinder provide a shortcut API To generating the large tree of handles needed for try/finally, but the JVM may not be able to optimize that tree as well as a purpose-built adapter. * Argument grouping operations in the middle of the argument list. JRuby has many signatures that vararg somewhere other than the end of the argument list, and the juggling required to do that logic in handles is complex: shift to-be-boxed args to end, box them, shift box back. Another point about these more complicated forms: they're ESPECIALLY slow early in execution, before LFs have been compiled to bytecode. * Implementation-specific inspection API. I know there are different ways to express a MH tree on different JVMs (e.g. J9) but it would still be a big help for me if there were a good way to get some debug-time structural information about a handle I'm using. Hidden API would be ok if it's not too hidden :-) That's off the top of my head. Others? - Charlie Hi, What would be nice to have is support for constant direct method handles and invokedynamic in Java language. We already have a syntax (thanks to lambdas) and target typing. What I'm asking for is for the following... constant direct method handles: class C { int someMethod(String s); } ... MethodHandle mh = C::someMethod(String); No pre-bound MHs, no overloaded method resolution. Just plain compilation to MH constant load. invokedynamic: There already was support for this in JSR292 prototype. Why was it removed? If for nothing else, this would simplify writing tests and experiments. Currently all we have is bytecode spinning with tools like ASM. Regards, Peter ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
Thanks Peter - I'd like that too. Another benefit is that it would (presumably) force IDEs to support the syntax so that you could this reference to, in your example, someMethod(). We had a problem in our large-ish codebase with methods that looked unused to the IDE (and hence the casual reader), but were in fact peeled off for use as MHs. We solved the is-it-used-or-not problem by switching to a helper lookup class which works off a @MethodHandle annotation. It's still not very navigable though, because the lookup referencing ultimately goes by method name strings and so isn't visible to the IDE. George On Mon, Mar 9, 2015 at 1:38 PM, Peter Levart peter.lev...@gmail.com wrote: On 02/26/2015 01:02 AM, Charles Oliver Nutter wrote: After talking with folks at the Jfokus VM Summit, it seems like there's a number of nice-to-have and a few need-to-have features we'd like to see get into java.lang.invoke. Vladimir suggested I start a thread on these features. A few from me: * A loop handle :-) Given a body and a test, run the body until the test is false. I'm guessing there's a good reason we don't have this already. * try/finally as a core atom of MethodHandles API. Libraries like invokebinder provide a shortcut API To generating the large tree of handles needed for try/finally, but the JVM may not be able to optimize that tree as well as a purpose-built adapter. * Argument grouping operations in the middle of the argument list. JRuby has many signatures that vararg somewhere other than the end of the argument list, and the juggling required to do that logic in handles is complex: shift to-be-boxed args to end, box them, shift box back. Another point about these more complicated forms: they're ESPECIALLY slow early in execution, before LFs have been compiled to bytecode. * Implementation-specific inspection API. I know there are different ways to express a MH tree on different JVMs (e.g. J9) but it would still be a big help for me if there were a good way to get some debug-time structural information about a handle I'm using. Hidden API would be ok if it's not too hidden :-) That's off the top of my head. Others? - Charlie Hi, What would be nice to have is support for constant direct method handles and invokedynamic in Java language. We already have a syntax (thanks to lambdas) and target typing. What I'm asking for is for the following... constant direct method handles: class C { int someMethod(String s); } ... MethodHandle mh = C::someMethod(String); No pre-bound MHs, no overloaded method resolution. Just plain compilation to MH constant load. invokedynamic: There already was support for this in JSR292 prototype. Why was it removed? If for nothing else, this would simplify writing tests and experiments. Currently all we have is bytecode spinning with tools like ASM. Regards, Peter ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
On 03/08/2015 01:38 PM, Jochen Theodorou wrote: Am 08.03.2015 12:16, schrieb Remi Forax: [...] You need to restrict the set of method handles that are allowed to be installed inside a CallSite corresponding to such kind of invokedynamic. Something like: you can do any transformations you want on the arguments but not on the receiver (no drop, no permute, no filter) and then you need to call unconditionally a method handle created using findSpecial on the super class. unconditionally, like no guard? @Jochen: There can be GWT, but it can only select one of target/fallback that are both from the restricted set of MHs. @Remi: That was my thinking too. I even started a proof-of-concept. The idea is as follows: 1) let verifier treat an invokedynamic super-init using a special system provided bootstrap method in the same way as it treats invokespecial super.init, meaning that such chaining to super constructor passes the verification 2) this system provided bootstrap method takes the standard parameters: - caller Lookup - invoked method name - invoked method type - and an additional parameter, a MethodHandle of a static method that is supplied by language runtime and has the purpose to build a MH chain that dispatches the invocation to one of candidate super constructors selected by system provided bootstrap method I started a proof-of-concept that tries to implement a similar setup, but instead of invoking super constructor, it invokes one of accessible superclass instance methods that have the same name as dynamically invoked name - this should translate to constructors nicely, I think. Here's a preview of what I'm trying to do: http://cr.openjdk.java.net/~plevart/misc/invoke.Story/Story.java The Story.superInvokeBootstrap is an example of such system provided bootstrap method that is trusted to do the right thing and treated by verifier in a special way. It prepares a set of target method handles and wraps them in wrapper objects that provide an API similar to MethodHandle/MethodHandles (Story.MH/Story) but restricted in a way that allows only transformations that maintain an invariant where the 1st argument (the target of instance method) is left untouched and invoking any of derived MHs results in either throwing an exception or invokes one of the provided candidate MHs. The language runtime must therefore provide a static MH-chain building method similar to the following: public static Story.MH buildDispatch( MethodHandles.Lookup callerLookup, String methodName, // the invoked method name MethodType methodType, // the invoked type Story story, ListStory.MH candidates) { ... } which uses Story.MH/Story API in a way that MethodHandle/MethodHandles API is used to construct and return a Story.MH that does the dispatch based on all arguments. So Jochen, can you point me to an algorithm used for MultiMethods dispatch in Groovy and I'll try to prepare a proof-of-concept that uses it? Regards, Peter bye Jochen ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
On 03/08/2015 12:56 AM, Peter Levart wrote: On 03/07/2015 02:53 PM, Remi Forax wrote: On 03/07/2015 06:31 AM, John Rose wrote: [...] (I wish we had a similar candidate for invokespecial/super. That is badly twisted around the verifier.) One way to solve the problem is to consider that invokedynamic init is a special 'bytecode' for the verifier and that the verifier will allow to call it on an non-initialized reference and consider the reference as been initialized after that. Rémi As I understand the problem is security-related. By mandating that each constructor chains to a super constructor (or one of sibling constructors that finally calls super), a class can rely on the fact that it's state is initialized using one of accessible constructors. The verifier makes sure there can be no subclass that doesn't call one of super constructors as part of object construction. Some security-related idioms are based on that guarantee. For example: services looked-up using ServiceLoader typically use an abstract class as the service interface that implementations must extend. The abstract class constructor checks for permissions. Each service implementation subclass must call the super constructor and the permission check in it has the subclass code on the call-stack when making a permission check. So how can verifier be sure that invokedynamic super-init eventually invokes one of the super constructors? Yes, right, I was expected something like this. You need to restrict the set of method handles that are allowed to be installed inside a CallSite corresponding to such kind of invokedynamic. Something like: you can do any transformations you want on the arguments but not on the receiver (no drop, no permute, no filter) and then you need to call unconditionally a method handle created using findSpecial on the super class. Peter Rémi ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
Am 08.03.2015 12:16, schrieb Remi Forax: [...] You need to restrict the set of method handles that are allowed to be installed inside a CallSite corresponding to such kind of invokedynamic. Something like: you can do any transformations you want on the arguments but not on the receiver (no drop, no permute, no filter) and then you need to call unconditionally a method handle created using findSpecial on the super class. unconditionally, like no guard? bye Jochen -- Jochen blackdrag Theodorou - Groovy Project Tech Lead blog: http://blackdragsview.blogspot.com/ german groovy discussion newsgroup: de.comp.lang.misc For Groovy programming sources visit http://groovy-lang.org ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
On 03/07/2015 06:31 AM, John Rose wrote: [...] (I wish we had a similar candidate for invokespecial/super. That is badly twisted around the verifier.) One way to solve the problem is to consider that invokedynamic init is a special 'bytecode' for the verifier and that the verifier will allow to call it on an non-initialized reference and consider the reference as been initialized after that. Rémi ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
(Even here there are non-universalities; what if the cleanup wants to see the return value and/or the thrown exception? Should it take those as one or two leading arguments?) Probably, it's too much. Result-agnostic cleanup logic looks like the cleanest solution from API perspective. There are catchException and foldArguments/filterReturnValue to combine with. Or am I missing something? As Charlie has explained, try/finally is *just barely* doable with the existing API points, as you note. I think we want a composite API point which can be readily compiled down to the natural JVM operations, instead of requiring the user to emit a complex pattern. The complex pattern will probably not be recognizable, and so won't be compiled down to the simple JVM operations. Don't get me wrong, I'm not against MHs.tryFinally. Quite the contrary :-) What I talked about is result-agnostic cleanup vs result/exceptional result passing into cleanup: MHs.tryFinally(target, cleanup)(*a) { try { return target(*a); } finally { cleanup(*a); } } vs MHs.tryFinally(target, cleanup)(*a) { Either?,Throwable ret; try { ret = new Left(target(*a)); } catch(Throwable e) { ret = new Right(e); } finally { cleanup(ret, *a); } } My point was that additional flexibility is probably too much, since it can be achieved using other combinators. Best regards, Vladimir Ivanov ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
On 03/07/2015 02:53 PM, Remi Forax wrote: On 03/07/2015 06:31 AM, John Rose wrote: [...] (I wish we had a similar candidate for invokespecial/super. That is badly twisted around the verifier.) One way to solve the problem is to consider that invokedynamic init is a special 'bytecode' for the verifier and that the verifier will allow to call it on an non-initialized reference and consider the reference as been initialized after that. Rémi As I understand the problem is security-related. By mandating that each constructor chains to a super constructor (or one of sibling constructors that finally calls super), a class can rely on the fact that it's state is initialized using one of accessible constructors. The verifier makes sure there can be no subclass that doesn't call one of super constructors as part of object construction. Some security-related idioms are based on that guarantee. For example: services looked-up using ServiceLoader typically use an abstract class as the service interface that implementations must extend. The abstract class constructor checks for permissions. Each service implementation subclass must call the super constructor and the permission check in it has the subclass code on the call-stack when making a permission check. So how can verifier be sure that invokedynamic super-init eventually invokes one of the super constructors? Peter ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
* try/finally as a core atom of MethodHandles API. Libraries like invokebinder provide a shortcut API To generating the large tree of handles needed for try/finally, but the JVM may not be able to optimize that tree as well as a purpose-built adapter. I agree there. We should put this in. MHs.tryFinally(target, cleanup)(*a) = { try { return target(*a); } finally { cleanup(*a); } } (Even here there are non-universalities; what if the cleanup wants to see the return value and/or the thrown exception? Should it take those as one or two leading arguments?) Probably, it's too much. Result-agnostic cleanup logic looks like the cleanest solution from API perspective. There are catchException and foldArguments/filterReturnValue to combine with. Or am I missing something? * Implementation-specific inspection API. I know there are different ways to express a MH tree on different JVMs (e.g. J9) but it would still be a big help for me if there were a good way to get some debug-time structural information about a handle I'm using. Hidden API would be ok if it's not too hidden :-) Idea of the day: An ASM-like library for method handles. Make a MethodHandleReader which can run a visitor over the MH. The ops of the visitor would be a selection of public MH operations like filter, collect, spread, lookup, etc. Also ASM-like, the library would have a MethodHandleWriter would could be hooked up with the reader to make filters. Not sure how useful it would be for an API user. I find it hard to recover MH transformation chains from MethodHandle trees, since the representation we use is quite low-level. They end up as trees of BoundMethodHandles and you can guess the intention only by carefully inspecting LambdaForms. You can try it yourself by turning on -Djava.lang.invoke.MethodHandle.DEBUG_NAMES=true and calling MethodHandle.toString() on a method handle of your choice. Best regards, Vladimir Ivanov ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
In other words, what we discussed was how do we see lightweight bytecode loading functionality in java.lang.invoke API. We don't want to make Unsafe.defineAnonymousClass() part of public API, but consider moving forward on that front providing lightweight code loading machinery. Best regards, Vladimir Ivanov On 3/3/15 2:12 PM, Marcus Lagergren wrote: At the VM language summit at JFokus 2015, we discussed having ways to get new dynamic language functions into the JVM without having to resort to generating a class wrapping their byte code. A class is currently the smallest possible compilation unit for the JVM, and its installation carries various overheads. Installing a non-anonymous class, as a lot of our classes need to be, for security reasons, also involve synchronizing on the system dictionary, and it seems that runtime has just given up on fixing that particular bottleneck [1] (I don’t agree at all with the conclusions in the CR). Currently, in Nashorn, whenever we regenerate a method due to a failed assumption or type specialization, we need to generate a new byte code method, wrap it in a synthetic class created for just that purpose, and then installing the class. When John and Vladimir were over in Stockholm we discussed a “magic” combinator that basically would allow you to create your own collection of MethodHandles for code versions of a callsite. Combined with constant pool indexes it would allow code installation without going through classes. New could would mean adding a {MethodHandle, ConstantPoolData} tuple to a particular callsite’s representation. /M [1] https://bugs.openjdk.java.net/browse/JDK-8046708 On 26 Feb 2015, at 13:42, MacGregor, Duncan (GE Energy Management) duncan.macgre...@ge.com wrote: MH.spreadArguments would certainly be useful from my point of view. We have many cases where we need to take a trailing argument array and turn it into some arguments, and array contain the remainder. This involves a depressing amount of shuffling at the moment, and should be better. On 26/02/2015 02:29, John Rose john.r.r...@oracle.com wrote: On Feb 25, 2015, at 4:02 PM, Charles Oliver Nutter head...@headius.com wrote: After talking with folks at the Jfokus VM Summit, it seems like there's a number of nice-to-have and a few need-to-have features we'd like to see get into java.lang.invoke. Vladimir suggested I start a thread on these features. A few from me: * A loop handle :-) Given a body and a test, run the body until the test is false. I'm guessing there's a good reason we don't have this already. A few reasons: 1. You can code your own easily. 2. There's no One True Loop the way there is a One True If. The run until test is false model assumes all the real work is done with side-effects, which are off-center from the MH model. 3. A really clean looping mechanism probably needs a sprinkle of tail call optimization. I'm not saying that loops should never have side effects, but I am saying that a loop mechanism should not mandate them. Maybe this is general enough: MHs.loop(init, predicate, body)(*a) = { let i = init(*a); while (predicate(i, *a)) { i = body(i, *a); } return i; } ...where the type of i depends on init, and if init returns void then you have a classic side-effect-only loop. * try/finally as a core atom of MethodHandles API. Libraries like invokebinder provide a shortcut API To generating the large tree of handles needed for try/finally, but the JVM may not be able to optimize that tree as well as a purpose-built adapter. I agree there. We should put this in. MHs.tryFinally(target, cleanup)(*a) = { try { return target(*a); } finally { cleanup(*a); } } (Even here there are non-universalities; what if the cleanup wants to see the return value and/or the thrown exception? Should it take those as one or two leading arguments?) * Argument grouping operations in the middle of the argument list. JRuby has many signatures that vararg somewhere other than the end of the argument list, and the juggling required to do that logic in handles is complex: shift to-be-boxed args to end, box them, shift box back. We now have MHs.collectArguments. Do you want MHs.spreadArguments to reverse the effect? Or is there something else I'm missing? Another point about these more complicated forms: they're ESPECIALLY slow early in execution, before LFs have been compiled to bytecode. * Implementation-specific inspection API. I know there are different ways to express a MH tree on different JVMs (e.g. J9) but it would still be a big help for me if there were a good way to get some debug-time structural information about a handle I'm using. Hidden API would be ok if it's not too hidden :-) Idea of the day: An ASM-like library for method handles. Make a MethodHandleReader which can run a visitor over the MH. The ops of the visitor would be a selection of public MH operations like filter, collect, spread, lookup, etc. Also
Re: What can we improve in JSR292 for Java 9?
From Valdimir We don't want to make Unsafe.defineAnonymousClass() part of public API, but consider moving forward on that front providing lightweight code loading machinery. While I used defineAnno for awhile currently I just use defineClass in my own class loader. My only concern is that perhaps its slow as its probably checking security, validating byte codes etc. All things that don't add any value in my app. I did originally like the ability to patch the constants but found just making constant callsites just as easy. Again maybe not as efficient. Seems like a class loader that specializes in dynamic languages would be a way to hide and localize the unsafe code while maintaining ease of use. I am all for low level incremental improvements at this point. I am concerned about what seems to be a focus on specific guest language support enhancements. mark ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
On Mar 6, 2015, at 5:30 AM, Vladimir Ivanov vladimir.x.iva...@oracle.com wrote: * try/finally as a core atom of MethodHandles API. Libraries like invokebinder provide a shortcut API To generating the large tree of handles needed for try/finally, but the JVM may not be able to optimize that tree as well as a purpose-built adapter. I agree there. We should put this in. MHs.tryFinally(target, cleanup)(*a) = { try { return target(*a); } finally { cleanup(*a); } } (Even here there are non-universalities; what if the cleanup wants to see the return value and/or the thrown exception? Should it take those as one or two leading arguments?) Probably, it's too much. Result-agnostic cleanup logic looks like the cleanest solution from API perspective. There are catchException and foldArguments/filterReturnValue to combine with. Or am I missing something? As Charlie has explained, try/finally is *just barely* doable with the existing API points, as you note. I think we want a composite API point which can be readily compiled down to the natural JVM operations, instead of requiring the user to emit a complex pattern. The complex pattern will probably not be recognizable, and so won't be compiled down to the simple JVM operations. * Implementation-specific inspection API. I know there are different ways to express a MH tree on different JVMs (e.g. J9) but it would still be a big help for me if there were a good way to get some debug-time structural information about a handle I'm using. Hidden API would be ok if it's not too hidden :-) Idea of the day: An ASM-like library for method handles. Make a MethodHandleReader which can run a visitor over the MH. The ops of the visitor would be a selection of public MH operations like filter, collect, spread, lookup, etc. Also ASM-like, the library would have a MethodHandleWriter would could be hooked up with the reader to make filters. Not sure how useful it would be for an API user. I find it hard to recover MH transformation chains from MethodHandle trees, since the representation we use is quite low-level. They end up as trees of BoundMethodHandles and you can guess the intention only by carefully inspecting LambdaForms. Yes, that's probably the downside. But (remaining hopeful) even if decoding the LF will not recover the original MH transformation chain, it may recover an *equivalent* one. You can try it yourself by turning on -Djava.lang.invoke.MethodHandle.DEBUG_NAMES=true and calling MethodHandle.toString() on a method handle of your choice. That's the first thing for a customer to try. — John ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
On Mar 4, 2015, at 2:47 PM, Charles Oliver Nutter head...@headius.com wrote: Busy week, finally circling back to this thread... On Wed, Feb 25, 2015 at 8:29 PM, John Rose john.r.r...@oracle.com wrote: * A loop handle :-) Given a body and a test, run the body until the test is false. I'm guessing there's a good reason we don't have this already. A few reasons: 1. You can code your own easily. I can't code one that will specialize for every call path, though, unless I generate a new loop body for every call path. Good point; if we had an intrinsic we could specialize this more readily, across more degrees of freedom. OTOH: What you are proposing is a workaround for a basic weakness in our system which we must eventually address. There is no reason why you shouldn't be able to write a fully generic algorithm (generic across calling sequences) and get good performance for each distinct instance of the algorithm. (Cue rants about the Loop Customization Problem.) But, as with try/finally, if there is a common use case which can only be expressed in a strained fashion using existing API points, we should provide a more natural notation for that use case; both users and JVM will find it easier to DTRT. 2. There's no One True Loop the way there is a One True If. The run until test is false model assumes all the real work is done with side-effects, which are off-center from the MH model. This I can appreciate. My mental model of MHs started to trend toward a general-purpose IR, and I believe if I had some sort of backward branch it could be that. But I understand if that's the wrong conceptual model, and I realize now that MHs are basically call stack adapters with a bit of forward branching thrown in. Good; you are hitting me where it counts, in the bytecodes. It does feel like there's a need for better representation of branch-joining or phi or whatever you want to call it, though. More details on this, please? The tail-part of a GWT is a phi. Or are you talking about phis on back-edges? I agree we need a loop, if only because we want to compile them to (the equivalent of) backwards-branching bytecodes. It's the same argument as for try/finally. 3. A really clean looping mechanism probably needs a sprinkle of tail call optimization. I'm not saying that loops should never have side effects, but I am saying that a loop mechanism should not mandate them. Maybe this is general enough: MHs.loop(init, predicate, body)(*a) = { let i = init(*a); while (predicate(i, *a)) { i = body(i, *a); } return i; } ...where the type of i depends on init, and if init returns void then you have a classic side-effect-only loop. Ahh yes, this makes sense. If it were unrolled, it would simply be a series of folds and drops as each iteration through the body modified the condition in some way. So then we just need it to work without unrolling. Glad you like it. This looks like a candidate, then. (I wish we had a similar candidate for invokespecial/super. That is badly twisted around the verifier.) My silly use case for this would be to emit simple expressions entirely as a MH chain, so we'd get the benefit of MH optimizations without generating our own bytecode (and with forced inlining and perhaps a richer semantic representation than just bytecode). It's not a very compelling case, of course, since I could just emit bytecode too. If it's a common use case, even if you could do it with bytecode spinning, it is reasonable to support a MH version, to help users avoid the friction of switching notations between MHs and bytecodes. * try/finally as a core atom of MethodHandles API. Libraries like invokebinder provide a shortcut API To generating the large tree of handles needed for try/finally, but the JVM may not be able to optimize that tree as well as a purpose-built adapter. I agree there. We should put this in. MHs.tryFinally(target, cleanup)(*a) = { try { return target(*a); } finally { cleanup(*a); } } (Even here there are non-universalities; what if the cleanup wants to see the return value and/or the thrown exception? Should it take those as one or two leading arguments?) In InvokeBinder, the finally is expected to require no additional arguments compared to the try body, since that was the use case I needed. Good to know. We start from what we know we need. You bring up a good point...and perhaps the built-in JSR292 tryFinally should take *two* handles: one for the exceptional path (with exception in hand) and one for the non-exceptional path (with return value in hand)? The exceptional path would be expected to return the same type as the try body or re-raise the exception. The non-exceptional path would be expected to return void. Yup. Or, the exception path could be a single MH that takes a union argument of U(normal-value, exception) and returns the expected value. In the JVM a
Re: What can we improve in JSR292 for Java 9?
Am 05.03.2015 11:09, schrieb Peter Levart: On 03/05/2015 04:09 AM, Jochen Theodorou wrote: [...] public class Foo { public Foo(String s, Integer i){} public Foo(Integer s, Object o){} } public class Bar extends Foo { public Bar(def a, def b) { super(a,b) // call with runtime types here } } I cannot express super(a,b) using method handles [...] Here's an idea. Let Groovy compile Foo to the following Java equivalent: public class Foo { public Foo(Onject p1, Object p2) { super(); invokedynamic _init(p1, p2); } private void _init(String s, Integer i) { // the body of constructor for (String, Integer) } private void _init(Integer s, Object o) { // the body of constructor for (Integer, Object) } } ok, sorry that I did not mention this contraint... Bar is in Groovy, Foo is written in Java. Meaning we have no control over Foo. Take for example BigDecimal instead of Foo. [...] So what we might need (also for other purposes like de-serialization) is a special kind of private void instance initialization methods that are treated specially by verifier and javac. The rules for such methods could be as follows: - they are treated like constructors regarding assignment to final instance fields - they can not be called from normal code except from constructors of the same class that calls: super constructor followed by a call to one of those special initialization methods. For example: public class Bar extends Foo { private final int val; public Bar(String s, int val) { super(s); // call to special @init method can only appear immediately after call to super constructor (verifier checked) initVal(val); // ...together they have the effect of calling any this(...) constructor // ...so this is not allowed by javac this.val = 42; } @init private void initVal(int val) { this.val = val; // allowed and required (like in constructor) } public void normalMethod() { initVal(42); // not allowed by javac or verifier } ... Those special @init methods could be invoked using reflection with overridden access checks (setAccessible(true)) and looked up as method handles using privileged Lookup only. hmm... might be an idea worth of investigation. I don't know if it can be done, because of the verifier. The biggest problem in the current logic is that you basically do an invokespecial with a kind of class reference loaded by an aload 0. But this is not a normal this, like you normally have. The sole purpose of it (afaik) is to call the super constructor. Any other call on this will cause problems. Your idea would, if we don't want to change the verifier, thus require an initialized class instance on the stack... which we won't get before calling super. But calling super is basically what we want to do. Btw, besides the special logic for final assignment... not all versions enforce that in the JVM. Reflection and Unsafe are often ways around that. But there is also the special logic for final fields and concurrent publication in the current memory model to consider here. I would not want to loose that. There is of course that internal deserialization logic Java has to create a blank instance. Maybe that could be used to have a kind of valid object to make calls on... But I am really not sure about the implications of that. In the end I really wonder if it is possible to implement these kinds of things without a change to the Verifier. If the Verifier can be tweaked a bit, then there should be a solution bye blackdrag -- Jochen blackdrag Theodorou - Groovy Project Tech Lead blog: http://blackdragsview.blogspot.com/ german groovy discussion newsgroup: de.comp.lang.misc For Groovy programming sources visit http://groovy-lang.org ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
On 03/05/2015 04:09 AM, Jochen Theodorou wrote: Am 04.03.2015 23:50, schrieb Charles Oliver Nutter: On Thu, Feb 26, 2015 at 4:27 AM, Jochen Theodorou blackd...@gmx.org wrote: my biggest request: allow the call of a super constructor (like super(foo,bar)) using MethodHandles an have it understood by the JVM like a normal super constructor call... same for this(...) Just so I understand...the problem is that unless you can get a Lookup that can do the super call from Java (i.e. from within a subclass), you can't get a handle that can do the super call, right? And you can't do that because the method bodies might not be emitted into a natural subclass of the super class? Not fully right. Sorry for the possibly false flag with the permissions from the runtime. I am in a natural subclass. Take this pseudoexample: public class Foo { public Foo(String s, Integer i){} public Foo(Integer s, Object o){} } public class Bar extends Foo { public Bar(def a, def b) { super(a,b) // call with runtime types here } } I cannot express super(a,b) using method handles, even if I did know the types at compiletime (in which case I would not need invokedynamic anyway). Sure, there is http://docs.oracle.com/javase/7/docs/api/java/lang/invoke/MethodHandles.Lookup.html#findSpecial%28java.lang.Class,%20java.lang.String,%20java.lang.invoke.MethodType,%20java.lang.Class%29 but this is for calling methods in a invokespecial like manner, init is excluded here as stated in the comment. The comment refers to findConsructor for calls to constructors, and indeed we have invokeSpecial calls there as well, but those are for what in Java is new, and are different from calls to a super constructor. See also http://mail.openjdk.java.net/pipermail/mlvm-dev/2012-June/004650.html from almost 3 years ago here on the list. Remi gives here a good reason why it probably cannot be done easily... in that or an even older thread someone (I think John) was talking about security. bye Jochen Hi Jochen, Here's an idea. Let Groovy compile Foo to the following Java equivalent: public class Foo { public Foo(Onject p1, Object p2) { super(); invokedynamic _init(p1, p2); } private void _init(String s, Integer i) { // the body of constructor for (String, Integer) } private void _init(Integer s, Object o) { // the body of constructor for (Integer, Object) } } ... a set of overloaded Groovy constructors with same arity could be emmited as: - a single constructor with Object typed arguments chaining to a super constructor with Object typed arguments using plain invokespecial - one private void _init method with declared types of arguments per Groovy constructor containing code of related Groovy constructor - an invokedynamic call in the single emitted constructor used for dispatching to appropriate _init method based on runtime types of arguments Only one thing would not work as expected in this setting: final fields could not be set in _init methods as verifier doesn't allow that. So what we might need (also for other purposes like de-serialization) is a special kind of private void instance initialization methods that are treated specially by verifier and javac. The rules for such methods could be as follows: - they are treated like constructors regarding assignment to final instance fields - they can not be called from normal code except from constructors of the same class that calls: super constructor followed by a call to one of those special initialization methods. For example: public class Bar extends Foo { private final int val; public Bar(String s, int val) { super(s); // call to special @init method can only appear immediately after call to super constructor (verifier checked) initVal(val); // ...together they have the effect of calling any this(...) constructor // ...so this is not allowed by javac this.val = 42; } @init private void initVal(int val) { this.val = val; // allowed and required (like in constructor) } public void normalMethod() { initVal(42); // not allowed by javac or verifier } ... Those special @init methods could be invoked using reflection with overridden access checks (setAccessible(true)) and looked up as method handles using privileged Lookup only. Regards, Peter ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
Julian wrote An open question that I have is that of facilitating the support of overloaded methods. It's typically something dynamically-typed languages struggle with, and doing it correctly *and* efficiently is not that pretty. Overloaded and multi methods have always been an interesting topic for me. I see two parts to this, the dispatch decision code and the view the programmer sees when coding. Of these the later seems to be the most difficult to envision in a object oriented dynamic language. This issue I see is that usually its not for a single type but for a family of types ( hierarchy) causing the pattern match to explode or the use of an isaFoo methods as part of the match. Another use would be in specialization of a call site based on argument types, perhaps using something like Truffle or LLVM jit to generate a specialized path. In this case one would want an efficient GWT which could pattern match on some instance var of each argument on the stack. There would probably only be a few of these per call site so one could do this today with a tree of GWTs and a pattern match test. So once again an efficient compiler/optimizer for GWT constructs would help along with counters and an handle to invoke to do the optimization. I share some other's concerns on performance impact of this. Perhaps the solution is to have a means to build an efficient GWT using Panama. regards mark___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
Busy week, finally circling back to this thread... On Wed, Feb 25, 2015 at 8:29 PM, John Rose john.r.r...@oracle.com wrote: * A loop handle :-) Given a body and a test, run the body until the test is false. I'm guessing there's a good reason we don't have this already. A few reasons: 1. You can code your own easily. I can't code one that will specialize for every call path, though, unless I generate a new loop body for every call path. 2. There's no One True Loop the way there is a One True If. The run until test is false model assumes all the real work is done with side-effects, which are off-center from the MH model. This I can appreciate. My mental model of MHs started to trend toward a general-purpose IR, and I believe if I had some sort of backward branch it could be that. But I understand if that's the wrong conceptual model, and I realize now that MHs are basically call stack adapters with a bit of forward branching thrown in. It does feel like there's a need for better representation of branch-joining or phi or whatever you want to call it, though. 3. A really clean looping mechanism probably needs a sprinkle of tail call optimization. I'm not saying that loops should never have side effects, but I am saying that a loop mechanism should not mandate them. Maybe this is general enough: MHs.loop(init, predicate, body)(*a) = { let i = init(*a); while (predicate(i, *a)) { i = body(i, *a); } return i; } ...where the type of i depends on init, and if init returns void then you have a classic side-effect-only loop. Ahh yes, this makes sense. If it were unrolled, it would simply be a series of folds and drops as each iteration through the body modified the condition in some way. So then we just need it to work without unrolling. My silly use case for this would be to emit simple expressions entirely as a MH chain, so we'd get the benefit of MH optimizations without generating our own bytecode (and with forced inlining and perhaps a richer semantic representation than just bytecode). It's not a very compelling case, of course, since I could just emit bytecode too. * try/finally as a core atom of MethodHandles API. Libraries like invokebinder provide a shortcut API To generating the large tree of handles needed for try/finally, but the JVM may not be able to optimize that tree as well as a purpose-built adapter. I agree there. We should put this in. MHs.tryFinally(target, cleanup)(*a) = { try { return target(*a); } finally { cleanup(*a); } } (Even here there are non-universalities; what if the cleanup wants to see the return value and/or the thrown exception? Should it take those as one or two leading arguments?) In InvokeBinder, the finally is expected to require no additional arguments compared to the try body, since that was the use case I needed. You bring up a good point...and perhaps the built-in JSR292 tryFinally should take *two* handles: one for the exceptional path (with exception in hand) and one for the non-exceptional path (with return value in hand)? The exceptional path would be expected to return the same type as the try body or re-raise the exception. The non-exceptional path would be expected to return void. We now have MHs.collectArguments. Do you want MHs.spreadArguments to reverse the effect? Or is there something else I'm missing? I want to be able to group arguments at any position in the argument list. Example: Incoming signature: (String foo, int a1, int a2, int a3, Object obj) Target signature: (String foo, int[] as, Object obj) ... without permuting arguments Idea of the day: An ASM-like library for method handles. Make a MethodHandleReader which can run a visitor over the MH. The ops of the visitor would be a selection of public MH operations like filter, collect, spread, lookup, etc. Also ASM-like, the library would have a MethodHandleWriter would could be hooked up with the reader to make filters. That would certainly cover it! I'd expect this to add: * MethodHandleVisitor interface of some kind * MethodHandle#accept(MethodHandleVisitor) or similar * MethodHandleType enum with all the base MH types (so we're not forcing all types into a static interface). - Charlie ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
On Thu, Feb 26, 2015 at 4:27 AM, Jochen Theodorou blackd...@gmx.org wrote: my biggest request: allow the call of a super constructor (like super(foo,bar)) using MethodHandles an have it understood by the JVM like a normal super constructor call... same for this(...) Just so I understand...the problem is that unless you can get a Lookup that can do the super call from Java (i.e. from within a subclass), you can't get a handle that can do the super call, right? And you can't do that because the method bodies might not be emitted into a natural subclass of the super class? - Charlie ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
Am 04.03.2015 23:50, schrieb Charles Oliver Nutter: On Thu, Feb 26, 2015 at 4:27 AM, Jochen Theodorou blackd...@gmx.org wrote: my biggest request: allow the call of a super constructor (like super(foo,bar)) using MethodHandles an have it understood by the JVM like a normal super constructor call... same for this(...) Just so I understand...the problem is that unless you can get a Lookup that can do the super call from Java (i.e. from within a subclass), you can't get a handle that can do the super call, right? And you can't do that because the method bodies might not be emitted into a natural subclass of the super class? Not fully right. Sorry for the possibly false flag with the permissions from the runtime. I am in a natural subclass. Take this pseudoexample: public class Foo { public Foo(String s, Integer i){} public Foo(Integer s, Object o){} } public class Bar extends Foo { public Bar(def a, def b) { super(a,b) // call with runtime types here } } I cannot express super(a,b) using method handles, even if I did know the types at compiletime (in which case I would not need invokedynamic anyway). Sure, there is http://docs.oracle.com/javase/7/docs/api/java/lang/invoke/MethodHandles.Lookup.html#findSpecial%28java.lang.Class,%20java.lang.String,%20java.lang.invoke.MethodType,%20java.lang.Class%29 but this is for calling methods in a invokespecial like manner, init is excluded here as stated in the comment. The comment refers to findConsructor for calls to constructors, and indeed we have invokeSpecial calls there as well, but those are for what in Java is new, and are different from calls to a super constructor. See also http://mail.openjdk.java.net/pipermail/mlvm-dev/2012-June/004650.html from almost 3 years ago here on the list. Remi gives here a good reason why it probably cannot be done easily... in that or an even older thread someone (I think John) was talking about security. bye Jochen -- Jochen blackdrag Theodorou - Groovy Project Tech Lead blog: http://blackdragsview.blogspot.com/ german groovy discussion newsgroup: de.comp.lang.misc For Groovy programming sources visit http://groovy-lang.org ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
Hello, I echo Jochen's griefs on constructors. There are cases of dynamic subclassing or bytecode wizardry where one would like to make that call with invokedynamic, and being forced into doing so with an invokespecial to not break the verifier rules is a big problem. Other than that having some support for intrinsic invocation counters on GWT branches would probably be very helpful when creating polymorphic-inline caches. One invalidation or periodic invalidation we could use those metrics to reorder the GWT, and possibly prune old ones. An open question that I have is that of facilitating the support of overloaded methods. It's typically something dynamically-typed languages struggle with, and doing it correctly *and* efficiently is not that pretty. - Julien ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
Marcus, On 3/03/2015 9:12 PM, Marcus Lagergren wrote: At the VM language summit at JFokus 2015, we discussed having ways to get new dynamic language functions into the JVM without having to resort to generating a class wrapping their byte code. A class is currently the smallest possible compilation unit for the JVM, and its installation carries various overheads. Installing a non-anonymous class, as a lot of our classes need to be, for security reasons, also involve synchronizing on the system dictionary, and it seems that runtime has just given up on fixing that particular bottleneck [1] (I don’t agree at all with the conclusions in the CR). Currently, in Nashorn, whenever we regenerate a method due to a failed assumption or type specialization, we need to generate a new byte code method, wrap it in a synthetic class created for just that purpose, and then installing the class. When John and Vladimir were over in Stockholm we discussed a “magic” combinator that basically would allow you to create your own collection of MethodHandles for code versions of a callsite. Combined with constant pool indexes it would allow code installation without going through classes. New could would mean adding a {MethodHandle, ConstantPoolData} tuple to a particular callsite’s representation. /M [1] https://bugs.openjdk.java.net/browse/JDK-8046708 This is a confidential JEP you just pointed people at. David - On 26 Feb 2015, at 13:42, MacGregor, Duncan (GE Energy Management) duncan.macgre...@ge.com wrote: MH.spreadArguments would certainly be useful from my point of view. We have many cases where we need to take a trailing argument array and turn it into some arguments, and array contain the remainder. This involves a depressing amount of shuffling at the moment, and should be better. On 26/02/2015 02:29, John Rose john.r.r...@oracle.com wrote: On Feb 25, 2015, at 4:02 PM, Charles Oliver Nutter head...@headius.com wrote: After talking with folks at the Jfokus VM Summit, it seems like there's a number of nice-to-have and a few need-to-have features we'd like to see get into java.lang.invoke. Vladimir suggested I start a thread on these features. A few from me: * A loop handle :-) Given a body and a test, run the body until the test is false. I'm guessing there's a good reason we don't have this already. A few reasons: 1. You can code your own easily. 2. There's no One True Loop the way there is a One True If. The run until test is false model assumes all the real work is done with side-effects, which are off-center from the MH model. 3. A really clean looping mechanism probably needs a sprinkle of tail call optimization. I'm not saying that loops should never have side effects, but I am saying that a loop mechanism should not mandate them. Maybe this is general enough: MHs.loop(init, predicate, body)(*a) = { let i = init(*a); while (predicate(i, *a)) { i = body(i, *a); } return i; } ...where the type of i depends on init, and if init returns void then you have a classic side-effect-only loop. * try/finally as a core atom of MethodHandles API. Libraries like invokebinder provide a shortcut API To generating the large tree of handles needed for try/finally, but the JVM may not be able to optimize that tree as well as a purpose-built adapter. I agree there. We should put this in. MHs.tryFinally(target, cleanup)(*a) = { try { return target(*a); } finally { cleanup(*a); } } (Even here there are non-universalities; what if the cleanup wants to see the return value and/or the thrown exception? Should it take those as one or two leading arguments?) * Argument grouping operations in the middle of the argument list. JRuby has many signatures that vararg somewhere other than the end of the argument list, and the juggling required to do that logic in handles is complex: shift to-be-boxed args to end, box them, shift box back. We now have MHs.collectArguments. Do you want MHs.spreadArguments to reverse the effect? Or is there something else I'm missing? Another point about these more complicated forms: they're ESPECIALLY slow early in execution, before LFs have been compiled to bytecode. * Implementation-specific inspection API. I know there are different ways to express a MH tree on different JVMs (e.g. J9) but it would still be a big help for me if there were a good way to get some debug-time structural information about a handle I'm using. Hidden API would be ok if it's not too hidden :-) Idea of the day: An ASM-like library for method handles. Make a MethodHandleReader which can run a visitor over the MH. The ops of the visitor would be a selection of public MH operations like filter, collect, spread, lookup, etc. Also ASM-like, the library would have a MethodHandleWriter would could be hooked up with the reader to make filters. ‹ John ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net
Re: What can we improve in JSR292 for Java 9?
On 03/04/2015 12:00 PM, Julien Ponge wrote: Hello, I echo Jochen's griefs on constructors. There are cases of dynamic subclassing or bytecode wizardry where one would like to make that call with invokedynamic, and being forced into doing so with an invokespecial to not break the verifier rules is a big problem. Other than that having some support for intrinsic invocation counters on GWT branches would probably be very helpful when creating polymorphic-inline caches. One invalidation or periodic invalidation we could use those metrics to reorder the GWT, and possibly prune old ones. When I was at Oracle office after JFokus, we discuss about how to implement a kind of lightweight safepointing. Charles need this to implement thread control in Ruby. I have the same kind of requirement, I want to be able to trigger a GC of an off-heap data structure managed by a C library (the code of the library is not thread safe :( ) so basically I want to be able to stop all the thread of my language to do a coordinate change. Technically it's not an invalidation because, the VM should not trash any code, just periodically enter in a code. It can not be implemented with a volatile flag because like a VM safepoint the check has to be inserted in most of the loops and the slowdown is noticeable, it can not be implemented with a SwitchPoint because it will trigger a deoptimization. The idea is to create another kind of MethodHandle that works like a SwitchPoint, but that let me switch it on and off several time. The VM will also be allowed to cluster several of these user safe points into one. The way to implement it in the VM is to do code patching using a non temporal move.to change the code that is executed and revert it back when necessary. Rémi An open question that I have is that of facilitating the support of overloaded methods. It's typically something dynamically-typed languages struggle with, and doing it correctly *and* efficiently is not that pretty. - Julien ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
Am 04.03.2015 12:00, schrieb Julien Ponge: Hello, I echo Jochen's griefs on constructors. There are cases of dynamic subclassing or bytecode wizardry where one would like to make that call with invokedynamic, and being forced into doing so with an invokespecial to not break the verifier rules is a big problem. thanks for the support Other than that having some support for intrinsic invocation counters on GWT branches would probably be very helpful when creating polymorphic-inline caches. One invalidation or periodic invalidation we could use those metrics to reorder the GWT, and possibly prune old ones. I am wondering about the performance loss if you do this with the current handles and store the counter in the callsite object. I mean with foldArguments and a void method you can make such an counter I think. An open question that I have is that of facilitating the support of overloaded methods. It's typically something dynamically-typed languages struggle with, and doing it correctly *and* efficiently is not that pretty. How would such support look like? A lookup with classes and the API matches that not exactly, but whatever is callable by Java rules? bye Jochen -- Jochen blackdrag Theodorou - Groovy Project Tech Lead blog: http://blackdragsview.blogspot.com/ german groovy discussion newsgroup: de.comp.lang.misc For Groovy programming sources visit http://groovy-lang.org ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
At the VM language summit at JFokus 2015, we discussed having ways to get new dynamic language functions into the JVM without having to resort to generating a class wrapping their byte code. A class is currently the smallest possible compilation unit for the JVM, and its installation carries various overheads. Installing a non-anonymous class, as a lot of our classes need to be, for security reasons, also involve synchronizing on the system dictionary, and it seems that runtime has just given up on fixing that particular bottleneck [1] (I don’t agree at all with the conclusions in the CR). Currently, in Nashorn, whenever we regenerate a method due to a failed assumption or type specialization, we need to generate a new byte code method, wrap it in a synthetic class created for just that purpose, and then installing the class. When John and Vladimir were over in Stockholm we discussed a “magic” combinator that basically would allow you to create your own collection of MethodHandles for code versions of a callsite. Combined with constant pool indexes it would allow code installation without going through classes. New could would mean adding a {MethodHandle, ConstantPoolData} tuple to a particular callsite’s representation. /M [1] https://bugs.openjdk.java.net/browse/JDK-8046708 On 26 Feb 2015, at 13:42, MacGregor, Duncan (GE Energy Management) duncan.macgre...@ge.com wrote: MH.spreadArguments would certainly be useful from my point of view. We have many cases where we need to take a trailing argument array and turn it into some arguments, and array contain the remainder. This involves a depressing amount of shuffling at the moment, and should be better. On 26/02/2015 02:29, John Rose john.r.r...@oracle.com wrote: On Feb 25, 2015, at 4:02 PM, Charles Oliver Nutter head...@headius.com wrote: After talking with folks at the Jfokus VM Summit, it seems like there's a number of nice-to-have and a few need-to-have features we'd like to see get into java.lang.invoke. Vladimir suggested I start a thread on these features. A few from me: * A loop handle :-) Given a body and a test, run the body until the test is false. I'm guessing there's a good reason we don't have this already. A few reasons: 1. You can code your own easily. 2. There's no One True Loop the way there is a One True If. The run until test is false model assumes all the real work is done with side-effects, which are off-center from the MH model. 3. A really clean looping mechanism probably needs a sprinkle of tail call optimization. I'm not saying that loops should never have side effects, but I am saying that a loop mechanism should not mandate them. Maybe this is general enough: MHs.loop(init, predicate, body)(*a) = { let i = init(*a); while (predicate(i, *a)) { i = body(i, *a); } return i; } ...where the type of i depends on init, and if init returns void then you have a classic side-effect-only loop. * try/finally as a core atom of MethodHandles API. Libraries like invokebinder provide a shortcut API To generating the large tree of handles needed for try/finally, but the JVM may not be able to optimize that tree as well as a purpose-built adapter. I agree there. We should put this in. MHs.tryFinally(target, cleanup)(*a) = { try { return target(*a); } finally { cleanup(*a); } } (Even here there are non-universalities; what if the cleanup wants to see the return value and/or the thrown exception? Should it take those as one or two leading arguments?) * Argument grouping operations in the middle of the argument list. JRuby has many signatures that vararg somewhere other than the end of the argument list, and the juggling required to do that logic in handles is complex: shift to-be-boxed args to end, box them, shift box back. We now have MHs.collectArguments. Do you want MHs.spreadArguments to reverse the effect? Or is there something else I'm missing? Another point about these more complicated forms: they're ESPECIALLY slow early in execution, before LFs have been compiled to bytecode. * Implementation-specific inspection API. I know there are different ways to express a MH tree on different JVMs (e.g. J9) but it would still be a big help for me if there were a good way to get some debug-time structural information about a handle I'm using. Hidden API would be ok if it's not too hidden :-) Idea of the day: An ASM-like library for method handles. Make a MethodHandleReader which can run a visitor over the MH. The ops of the visitor would be a selection of public MH operations like filter, collect, spread, lookup, etc. Also ASM-like, the library would have a MethodHandleWriter would could be hooked up with the reader to make filters. ‹ John ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
New could would mean adding a {MethodHandle, ConstantPoolData} tuple to a particular callsite?s representation. Now if we could add a item for the object reference used for the comparison in the test part of a GWT we would have a structure which could be used in a PIC (polymorphic inline cache) implementation. A PIC then would be this collection, a test method handle ( used to extract the comparison value from an object on the stack) , a fall back method handle, perhaps a overflow handle ( if the inlining is exceeded), and some attributes to inform the compiler of how to optimize ( e.g inline depth of the guards). Reflection and branch counters would be nice if we are expected to optimize externally ( order and size of inlining ). An interesting optimization is that often many objects share the same code, sometimes there is only one implementation for all objects. This essentially turns most megamorphic sites into monomorphic. Of course with a dynamic language its hard to guarantee this. But one could have an inverse PIC. One where its built with markers for the unique implementations and does a fast fallback to the common. Perhaps the solution to this is Panama and a local JIT, so maybe a callsite which wrappers an intrinsic like piece of could would do it. It would be nice if this level of detail was hidden by the jvm. As an aside the fast path slow path nomenclature is bothersome as most of the time they are all fast. Only the fallback which does a look up is slow. mark ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
MH.spreadArguments would certainly be useful from my point of view. We have many cases where we need to take a trailing argument array and turn it into some arguments, and array contain the remainder. This involves a depressing amount of shuffling at the moment, and should be better. On 26/02/2015 02:29, John Rose john.r.r...@oracle.com wrote: On Feb 25, 2015, at 4:02 PM, Charles Oliver Nutter head...@headius.com wrote: After talking with folks at the Jfokus VM Summit, it seems like there's a number of nice-to-have and a few need-to-have features we'd like to see get into java.lang.invoke. Vladimir suggested I start a thread on these features. A few from me: * A loop handle :-) Given a body and a test, run the body until the test is false. I'm guessing there's a good reason we don't have this already. A few reasons: 1. You can code your own easily. 2. There's no One True Loop the way there is a One True If. The run until test is false model assumes all the real work is done with side-effects, which are off-center from the MH model. 3. A really clean looping mechanism probably needs a sprinkle of tail call optimization. I'm not saying that loops should never have side effects, but I am saying that a loop mechanism should not mandate them. Maybe this is general enough: MHs.loop(init, predicate, body)(*a) = { let i = init(*a); while (predicate(i, *a)) { i = body(i, *a); } return i; } ...where the type of i depends on init, and if init returns void then you have a classic side-effect-only loop. * try/finally as a core atom of MethodHandles API. Libraries like invokebinder provide a shortcut API To generating the large tree of handles needed for try/finally, but the JVM may not be able to optimize that tree as well as a purpose-built adapter. I agree there. We should put this in. MHs.tryFinally(target, cleanup)(*a) = { try { return target(*a); } finally { cleanup(*a); } } (Even here there are non-universalities; what if the cleanup wants to see the return value and/or the thrown exception? Should it take those as one or two leading arguments?) * Argument grouping operations in the middle of the argument list. JRuby has many signatures that vararg somewhere other than the end of the argument list, and the juggling required to do that logic in handles is complex: shift to-be-boxed args to end, box them, shift box back. We now have MHs.collectArguments. Do you want MHs.spreadArguments to reverse the effect? Or is there something else I'm missing? Another point about these more complicated forms: they're ESPECIALLY slow early in execution, before LFs have been compiled to bytecode. * Implementation-specific inspection API. I know there are different ways to express a MH tree on different JVMs (e.g. J9) but it would still be a big help for me if there were a good way to get some debug-time structural information about a handle I'm using. Hidden API would be ok if it's not too hidden :-) Idea of the day: An ASM-like library for method handles. Make a MethodHandleReader which can run a visitor over the MH. The ops of the visitor would be a selection of public MH operations like filter, collect, spread, lookup, etc. Also ASM-like, the library would have a MethodHandleWriter would could be hooked up with the reader to make filters. ‹ John ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
Am 26.02.2015 01:02, schrieb Charles Oliver Nutter: After talking with folks at the Jfokus VM Summit, it seems like there's a number of nice-to-have and a few need-to-have features we'd like to see get into java.lang.invoke. Vladimir suggested I start a thread on these features. my biggest request: allow the call of a super constructor (like super(foo,bar)) using MethodHandles an have it understood by the JVM like a normal super constructor call... same for this(...) Because what we currently do is annoying and a major pita, plus it bloats the bytecode we have to produce. And let us better not talk about speed or the that small verifier change that made our hack unusable in several java update versions for 7 and 8. This has been denied in the past because of security reasons... And given that we need dynamic argument types to determine the constructor to be called, and since that we have to do a call from the runtime in the uncached case, I fully understand why this is not done... just... it would be nice to have a solution that does not require us doing basically a big switch table with several invokespecial calls bye Jochen -- Jochen blackdrag Theodorou - Groovy Project Tech Lead blog: http://blackdragsview.blogspot.com/ german groovy discussion newsgroup: de.comp.lang.misc For Groovy programming sources visit http://groovy-lang.org ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
On Feb 25, 2015, at 4:02 PM, Charles Oliver Nutter head...@headius.com wrote: After talking with folks at the Jfokus VM Summit, it seems like there's a number of nice-to-have and a few need-to-have features we'd like to see get into java.lang.invoke. Vladimir suggested I start a thread on these features. A few from me: * A loop handle :-) Given a body and a test, run the body until the test is false. I'm guessing there's a good reason we don't have this already. A few reasons: 1. You can code your own easily. 2. There's no One True Loop the way there is a One True If. The run until test is false model assumes all the real work is done with side-effects, which are off-center from the MH model. 3. A really clean looping mechanism probably needs a sprinkle of tail call optimization. I'm not saying that loops should never have side effects, but I am saying that a loop mechanism should not mandate them. Maybe this is general enough: MHs.loop(init, predicate, body)(*a) = { let i = init(*a); while (predicate(i, *a)) { i = body(i, *a); } return i; } ...where the type of i depends on init, and if init returns void then you have a classic side-effect-only loop. * try/finally as a core atom of MethodHandles API. Libraries like invokebinder provide a shortcut API To generating the large tree of handles needed for try/finally, but the JVM may not be able to optimize that tree as well as a purpose-built adapter. I agree there. We should put this in. MHs.tryFinally(target, cleanup)(*a) = { try { return target(*a); } finally { cleanup(*a); } } (Even here there are non-universalities; what if the cleanup wants to see the return value and/or the thrown exception? Should it take those as one or two leading arguments?) * Argument grouping operations in the middle of the argument list. JRuby has many signatures that vararg somewhere other than the end of the argument list, and the juggling required to do that logic in handles is complex: shift to-be-boxed args to end, box them, shift box back. We now have MHs.collectArguments. Do you want MHs.spreadArguments to reverse the effect? Or is there something else I'm missing? Another point about these more complicated forms: they're ESPECIALLY slow early in execution, before LFs have been compiled to bytecode. * Implementation-specific inspection API. I know there are different ways to express a MH tree on different JVMs (e.g. J9) but it would still be a big help for me if there were a good way to get some debug-time structural information about a handle I'm using. Hidden API would be ok if it's not too hidden :-) Idea of the day: An ASM-like library for method handles. Make a MethodHandleReader which can run a visitor over the MH. The ops of the visitor would be a selection of public MH operations like filter, collect, spread, lookup, etc. Also ASM-like, the library would have a MethodHandleWriter would could be hooked up with the reader to make filters. — John ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
On Feb 25, 2015, at 5:09 PM, Mark Roos mr...@roos.com wrote: I would like to see some form of PIC (polymorphic inline cache ) support that jits (inlines) well and transitions from mono to bi to multi(4,5) to mega nicely. What would be the starting points for such a thing? It seems to be like Charlie's MH builder, something that would be a library class. But maybe (like tryFinally) there are points where the library class has a hard time making its intentions clear to the JVM, because of API deficiencies. (Sorry this isn't more specific.) — John ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev
Re: What can we improve in JSR292 for Java 9?
I would like to see some form of PIC (polymorphic inline cache ) support that jits (inlines) well and transitions from mono to bi to multi(4,5) to mega nicely. The main thought would be to inline two or three with some way to reopt as the types/counters change. Of course my types are not java classes but are carried as a reference in a local var slot, so optimizing the test would be good. regards mark ___ mlvm-dev mailing list mlvm-dev@openjdk.java.net http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev