Re: A pure Java example with reflective access behaving differently with invokeWithArguments() compared to core reflection's invoke() (Re: Strange observation: MethodHandle.invokeWithArguments() would

[Peter Levart]

 the question would be: is this considered 
a bug that will be corrected? Would there be otherwise a solution 
possible in the bridge that could take care of situations like this, 
where the bridge has no context-information other than referring to 
Constructor, Method or Field objects with the supplied arguments?


A few coarse ideas to address this:

  * if the unreflect() method gets used then MH.invoke() should behave
like in the java.lang.reflect case; this means that the same
assumptions should govern MH.invoke(): reason being that if a
java.lang.reflect object gets unreflected, also the established
java.lang.reflect rules should keep working in this case for the
MH.invoke* in order to remain fully compatible with each other,
  o if a MH gets created without unreflect() then this would not
be necessary
  * the bridge could restrain itself to only use java.lang.reflect for
invocations on Java 8 and higher instead of java.lang.invoke
  o clearly, the current Java development efforts in this corner
are concentrated on java.lang.invoke, so it would be desirable
to switch from java.lang.reflect to java.lang.invoke which is
not possible as long as this problem (different behaviour in
j.l.r vs. j.l.i) persists
  * do not fix this, but document this as an incompatibility with the
ramification that existing programs in dynamically typed languages
need to be rewritten (e.g. in this case creating an Array object
from the List and supply that instead)
  * ... ?

---rony

P.S.: It would be great if Java implemented reflective invocations for 
dynamically typed languages as this would solve the problem for all 
such languages once and forever in a standard manner, rather than have 
every bridge implementor create his own implementation, which is 
challenging,  effortful and possibly error prone. In my case the 
assumptions that governed the original implementation for reflective 
Constructor, Method and Field access for over fifteen years got broken 
by Java 9, because j.l.r (notable 'setAccessible') changed in a 
fundamental aspect. If Java could be emplyoed instead, the Java 
supplied implementation would be able to adopt changes in the inner 
workings accordingly thereby insulating bridges from such details and 
problems.



On 08.06.2018 18:23, Peter Levart wrote:
I think what you found is a discrepancy between handling of varargs 
methods using reflection vs. method handle's .invokeWithArguments().


Reflection basically ignores the fact that some methods are varargs 
methods. It treats them exactly the same as if they had an array 
parameter (not a varargs parameter).


For example, Arrays.asList() is a varargs method:

    public static  List asList(T ... elements)

But reflection treats is exactly the same as the following method:

    public static  List asList(T[] elements)

That's because varargs were introduced to Java in Java 5 as a kind of 
compilation sugar, implemented as an array parameter, while 
reflection is an older beast and wasn't updated to behave differently 
with varargs methods.


MethodHandle(s) came later, in Java 7, and took into consideration 
varargs methods.


So where's the problem? When you say in Java:

    Method asListMethod = Arrays.class.getMethod("asList", 
Object[].class);

    String[] elements = {"a", "b", "c"};

    System.out.println(
    asListMethod.invoke(null, elements)
    );

You get: IllegalArgumentException: wrong number of arguments

Because Method.invoke() is also a varargs method:

    public Object invoke(Object obj, Object... args)

... so java compiler (in order to be backwards source compatible with 
pre-varargs methods) treats this invocation in a way that just passes 
the String[] elements to the invoke method via the args parameter 
without any conversion (because String[] is a subtype of Object[]). 
Mathod.invoke then treats elements of the args[] array as individual 
verbatim parameters to be passed to the method when invoking it. As 
reflection treats Arrays.asList() as a normal no-varargs method which 
accepts a single Objet[] parameter and args[] contains 3 elements to 
be passed to a method with 3 parameters, exception occurs.



But when you use method handles and .invokeWithArguments():

    Method asListMethod = Arrays.class.getMethod("asList", 
Object[].class);
    MethodHandle asListMH = 
MethodHandles.lookup().unreflect(asListMethod);


    String[] elements = {"a", "b", "c"};

    System.out.println(
    asListMH.invokeWithArguments(elements)
    );

... the following happens: MethodHandle.invokeWithArguments() is also 
a varargs method:


    public Object invokeWithArguments(Object... arguments)

... so java compiler (in order to be backwards source compatible with 
pre-varargs methods) treats this invocation in a way that just

Re: A pure Java example with reflective access behaving differently with invokeWithArguments() compared to core reflection's invoke() (Re: Strange observation: MethodHandle.invokeWithArguments() would

[Peter Levart]

Hi Rony,

I think what you found is a discrepancy between handling of varargs 
methods using reflection vs. method handle's .invokeWithArguments().


Reflection basically ignores the fact that some methods are varargs 
methods. It treats them exactly the same as if they had an array 
parameter (not a varargs parameter).


For example, Arrays.asList() is a varargs method:

    public static  List asList(T ... elements)

But reflection treats is exactly the same as the following method:

    public static  List asList(T[] elements)

That's because varargs were introduced to Java in Java 5 as a kind of 
compilation sugar, implemented as an array parameter, while reflection 
is an older beast and wasn't updated to behave differently with varargs 
methods.


MethodHandle(s) came later, in Java 7, and took into consideration 
varargs methods.


So where's the problem? When you say in Java:

    Method asListMethod = Arrays.class.getMethod("asList", 
Object[].class);

    String[] elements = {"a", "b", "c"};

    System.out.println(
    asListMethod.invoke(null, elements)
    );

You get: IllegalArgumentException: wrong number of arguments

Because Method.invoke() is also a varargs method:

    public Object invoke(Object obj, Object... args)

... so java compiler (in order to be backwards source compatible with 
pre-varargs methods) treats this invocation in a way that just passes 
the String[] elements to the invoke method via the args parameter 
without any conversion (because String[] is a subtype of Object[]). 
Mathod.invoke then treats elements of the args[] array as individual 
verbatim parameters to be passed to the method when invoking it. As 
reflection treats Arrays.asList() as a normal no-varargs method which 
accepts a single Objet[] parameter and args[] contains 3 elements to be 
passed to a method with 3 parameters, exception occurs.



But when you use method handles and .invokeWithArguments():

    Method asListMethod = Arrays.class.getMethod("asList", 
Object[].class);
    MethodHandle asListMH = 
MethodHandles.lookup().unreflect(asListMethod);


    String[] elements = {"a", "b", "c"};

    System.out.println(
    asListMH.invokeWithArguments(elements)
    );

... the following happens: MethodHandle.invokeWithArguments() is also a 
varargs method:


    public Object invokeWithArguments(Object... arguments)

... so java compiler (in order to be backwards source compatible with 
pre-varargs methods) treats this invocation in a way that just passes 
the String[] elements to the invokeWithArguments method via the single 
'arguments' parameter without any conversion (because String[] is a 
subtype of Object[]). MethodHandle.invokeWithArguments therefore takes 3 
array elements and tries to invoke the underlying asList method with 
them. It observes that Arrays.asList is a varargs method, so it wraps 
the 3 "wannabe parameters" with the Object[] and passes the array to the 
asList method as single parameter. The result of above code is therefore:


    [a, b, c]

If you want MethodHandle.invokeWithArguments() to treat "wannabe 
parameters" passed to it as verbatim parameters regardless of the 
variable/fixed arity of the method to be invoked, then you can transform 
the variable arity MethodHandle to a fixed arity MH:


    Method asListMethod = Arrays.class.getMethod("asList", 
Object[].class);


    String[] elements = {"a", "b", "c"};

    System.out.println(
    asListMethod.invoke(null, (Object) elements)
    );

    MethodHandle asListMH = 
MethodHandles.lookup().unreflect(asListMethod);

    asListMH = asListMH.asFixedArity();

    System.out.println(
    asListMH.invokeWithArguments((Object) elements)
    );

Here you have it. Both of the above invocations produce equal output:

    [a, b, c]
    [a, b, c]


What happens here is the following (will only describe the MH case - 
Method case is similar):


MethodHandle.invokeWithArguments() is a varargs method:

    public Object invokeWithArguments(Object... arguments)

... so java compiler this time wraps the '(Object) elements' value with 
an Object[] because the value is of Object type (which is not a subtype 
of Object[]) - no backwards compatibility needed here, varargs 
conversion kicks in in the javac. MethodHandle.invokeWithArguments 
therefore takes an array with 1 element (the element being 'elements' 
array) and tries to invoke the underlying asList method with it. This 
time it observes that the method is not a varargs method, (because 
MethodHandle was transformed to fixed arity method handle) so it passes 
the single 'elements' wannabee parameter to the Arrays.asList single 
Object[] parameter - this invocation works because fixed arity asList() 
takes Object[] and 'elements' is a String[]...


Hope this helps you understand what's going on.

Regards, Peter

On 06/08/18 12:07, Rony G. Flatscher wrote:

On 11.03.2018 20:22, Rony 

Re: ClassValue rooting objects after it goes away?

[Peter Levart]

Hi Charles,

On 03/02/2018 09:22 PM, Charles Oliver Nutter wrote:
Put it another way: does a static reference from a class to itself 
prevent that class from being garbage collected? Of course not. 
ClassValue is intended to be a way to inject pseudo-static data into 
either a class or a Class. Injecting that data, even if it has a 
reference back to the class, should not prevent the class from being 
collected.


And it should not with ClassValue either. The problem is when you 
associate values with "system" classes (such as Object, String, etc.). 
Such values become strongly reachable from Object.class, String.class, 
etc., respectively. It's like injecting an instance field into Class 
class and assigning Object.class.myField = myValue, String.class.myField 
= myValue, ... or equivalently injecting static fields into Object, 
String, etc class(es) and assigning Object.myField = myValue, 
String.myField = myValue, ...


This would not in itself be a problem if myValue was composed only of 
objects of system classes (like String, Integer, HashMap, ArrayList, 
etc..), but you probably have your own Ruby runtime classes to maintain 
the data structure behind myValue. And each of those custom objects 
maintains an implicit reference to its runtime class, which is a Ruby 
runtime class, loaded by whichever ClassLoader loaded the Ruby runtime, 
so the chain of strong references extends from system class loader to 
Ruby runtime class loader, effectively preventing it from being unloaded.


If you don't need to use ClassValue.remove() in your code, you could try 
using the following wrapper above ClassValue. The overhead of its get() 
method compared to ClassValue's should not be to big - just another 
lookup into a ThreadLocal and a reference comparison with null (in the 
general fast-path case):



/**
 * {@link ClassValue} makes mapped values strongly reachable from 
{@link Class}(es)
 * they are associated with while they are not reachable directly from 
{@link ClassValue}(s).
 * This is some times desirable, but other times not. This wrapper 
provides a
 * ClassValue-like API where the roles are reversed - associated values 
are not strongly
 * reachable from Class(es) but from WeakClassValue(s) they are 
associated with.

 *
 * @param  they type of associated values
 */
public abstract class WeakClassValue {

    // temporary holder of a Node computed in embedded 
cv.computeValue() so we
    // can later find out whether the embedded WeakReference has been 
installed into
    // ClassValueMap or not by comparing it to instance returned from 
cv.get()

    private static final ThreadLocal NODE_TL = new ThreadLocal<>();

    // head of doubly-linked list of nodes, keeping associated values 
reachable from

    // WeakClassValue instance
    private final Node head = new Node(null, null, null, null, null);

    // a single reference queue for all Node(s) to get enqueued when 
their referents

    // (Class instances) get GCed...
    private static final ReferenceQueue queue = new 
ReferenceQueue<>();


    // Node is a PhantomReference tracking reachability of associated 
Class.
    // it also holds the associated value, WeakReference to it, and 
prev/next pointers into the list

    private static final class Node extends PhantomReference {
    private final Node head;
    private Node prev, next;
    private Object value;
    WeakReference ref;

    Node(Class type, ReferenceQueue queue,
 Node head, Object value, WeakReference ref) {
    super(type, queue);
    this.value = value;
    this.ref = ref;
    if (head == null) {
    // constructing head node
    this.prev = this.next = this;
    this.head = null;
    } else {
    // link node into list anchored by head
    synchronized (head) {
    this.prev = head;
    this.next = head.next;
    this.head = head; // remember where we belong to
    head.next.prev = this;
    head.next = this;
    }
    }
    }

    // unlink the node from the list using right head as a lock
    void unlink() {
    synchronized (head) {
    prev.next = next;
    next.prev = prev;
    next = prev = null;
    value = null;
    ref.clear();
    ref = null;
    }
    }
    }

    // work is delegated to embedded ClassValue instance
    private final ClassValue cv = new ClassValue<>() {
    @SuppressWarnings("unchecked")
    @Override
    protected WeakReference computeValue(Class type) {
    T value = WeakClassValue.this.computeValue(type);
    WeakReference ref = new WeakReference<>(value);
    // link the value into a doubly-linked list rooted at
    // WeakClassValue 

Re: Isolated Methods JEP

[Peter Levart]

Hi Michael,


On 08/09/2016 02:05 PM, Peter Levart wrote:
In which case would lookup privileges be insufficient to define an 
isolated method? I would expect the privileges of the code in the 
isolated method to be checked when such method is executed and not 
when defining such method - lazily, like for normal methods.


A can see now that defining an isolated method in the context of a 
lookup class can only be allowed when the Lookup instance contains 
ALL_MODES in the mode bits as such method would have access to any 
members that a normal method defined in the lookup class has.


Regards, Peter

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: Isolated Methods JEP

[Peter Levart]

Hi Michael,

In the JEP's description you write:

   /"The loadCode method creates a method from the passed bytecode
   instructions and constants and returns a MethodHandle that can be
   used to call the method. The implementation of loadCode will take
   care of verification of the code to load.//"/

//
That's clear.

//

   /"This method is isolated from any class and behaves largely like a
   static method. The method handle resulting from a loadCode
   invocation is of the REF_static kind. It cannot be cracked via
   MethodHandles.Lookup.revealDirect().//"/


So there will be no class hosting this method? Not even the "caller" 
class of the Lookup instance that loadCode was invoked upon? What will 
be reported by Reflection.getCallerClass() invoked in a @CallerSensitive 
method invoked from the isolated method? Perhaps the following is a hint...


//

   /"The context for a method defined in this way is determined by the
   Lookup instance receiving the loadCode call. In case the lookup
   privileges are not sufficient, an exception will be thrown.//"

   /

The "context" meaning the caller context, including the caller class 
that appears in the stack trace of a call originating from an isolated 
method and is important for security decisions?


In which case would lookup privileges be insufficient to define an 
isolated method? I would expect the privileges of the code in the 
isolated method to be checked when such method is executed and not when 
defining such method - lazily, like for normal methods. In which case it 
is important which "caller" class the privileges are check against. This 
would most naturally be the "caller" class of the Lookup instance used 
to define the method. In all respects the code of such isolated method 
would appear to be defined in the lookup class except it would not be 
denotable symbolically - only through a MethodHandle. Analogous to 
VM-anonymous classes. So why not call such methods "anonymous methods" 
instead of isolated methods?



Regards, Peter


On 08/05/2016 06:33 PM, Michael Haupt wrote:

Dear all,

during the Indy workshop at JVMLS, a JEP draft supposed to enable the 
loading of methods in isolation was mentioned and discussed. This JEP 
draft is now public, and I'd like to invite The Interested Parties 
(you know who you are) to tear it apart in a constructive manner. :-)


https://bugs.openjdk.java.net/browse/JDK-8158765

Thanks,

Michael

--

Oracle 
Dr. Michael Haupt | Principal Member of Technical Staff
Phone: +49 331 200 7277 | Fax: +49 331 200 7561
OracleJava Platform Group | LangTools Team | Nashorn
Oracle Deutschland B.V. & Co. KG | Schiffbauergasse 14 | 14467 
Potsdam, Germany


ORACLE Deutschland B.V. & Co. KG | Hauptverwaltung: Riesstraße 25, 
D-80992 München

Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V. | Hertogswetering 
163/167, 3543 AS Utrecht, Niederlande

Handelsregister der Handelskammer Midden-Nederland, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher
Green Oracle  	Oracle is committed 
to developing practices and products that help protect the environment





___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: RFR: JDK-8031043 ClassValue's backing map should have a smaller initial size

[Peter Levart]

Hi,

Doug suggested to try another variant of resolving hash collisions in 
open-addressing table - using a secondary hash for increment value of 
consecutive probes. Since Java Objects provide only one hashCode() 
function, "secondary" hash is computed from primary hash by 
xorShift(hashCode), for example:


   r ^= r << 13; r ^= r >>> 17; r ^= r << 5;

Such hash is then made "odd" by or-ing with 1 and used as probe 
increment (multiplied by 2 when table uses two slots per entry such as 
in our case).


Adding this strategy to the mix in the simulator:

http://cr.openjdk.java.net/~plevart/misc/HibrydHashtable/OpenAddressingProbeSequence.java

Gives promising results with MethodType or Object keys (ClassValue keys 
are perfect and don't need this anyway):


http://cr.openjdk.java.net/~plevart/misc/HibrydHashtable/lpht_MethodType_probe_sequence.txt

The simulation also provides results of probe length distribution for 
linked-nodes table such as ConcurrentHashMap to compare.


Secondary hash as probe stride gives the best average probe length among 
open-addressing strategies and is not dependent very much on the 
insertion order (sorted vs. unsorted) of key's hashes, but it does not 
give best worst-case lookup performance (max. probe length). Worst-case 
lookup among open-addressing tables tried still belongs to quadratic 
probing in combination with sorted insertion order.


I created a secondary-hash probing implementation 
(LinearProbeHashtable3, cyan color in the graph) and compared its 
MethodType keys lookup performance with other variants created before 
and CHM:


http://cr.openjdk.java.net/~plevart/misc/HibrydHashtable/lpht_MethodType_bench.pdf


Unlike what simulator tells us about average probe length, in practice 
(on my i7-4771 CPU), secondary-hash probing does not excel. I think that 
it suffers from lack of locality of reference - it is not very nice to 
CPU cache. It might have been a better option in the past when CPU 
caches were not so sophisticated. Quadratic probing with sorted 
insertion seems to hit the sweet spot between optimizing the locality of 
reference and average probe length.


"What is a HybridHashtable?", you might ask, since it beats all others 
in the benchmark...


Here's the implementation:

http://cr.openjdk.java.net/~plevart/misc/HibrydHashtable/lpht_MethodType_bench_src.tgz

The above simulator shows that for MethodType or Object keys, average 
probe length in linked-nodes tables such as CHM is half the average 
probe length of quadratic probing open-addressing table (~0.25 vs. 
~0.5), but quadratic probing still beats linked-nodes table because it 
has better locality of reference. What simulator also shows is that ~80% 
of linked-nodes buckets contains a single entry (probe-length[0] 
histogram). HybridHashtable looks like open-addressing table for ~80% of 
entries and like linked-nodes table for the rest of them:


 * A lone entry (K1, V1) in hybrid table:
 *
 *+-- (K1.hashCode() * 2) % table.length
 *|
 * table  v
 * -+++-
 *  | K1 | V1 |
 * -+++-
 *
 * When a second entry (K2, V2) is inserted that happens to hash to the
 * same "bucket", above situation is expanded into:
 *
 * table
 * -+++-
 *  | K1 | *  |
 * -+++-
 * |
 * v Node
 *   ++++
 *   | V1 | K2 | V2 |
 *   ++++
 *
 * Similarly, when a third entry is inserted, we get:
 *
 * table
 * -+++-
 *  | K1 | *  |
 * -+++-
 * |
 * v Node
 *   ++++
 *   | V1 | K2 | *  |
 *   ++++
 *   |
 *   v Node
 * ++++
 * | V2 | K3 | V3 |
 * ++++
 *
 * ...and so on.

Such arrangement combines the benefits of locality of reference inherent 
to open-addressing tables with minimal average number of false probes 
which is a property of linked-nodes tables. It seems that in practice, 
it beats other strategies tried here when lookup performance is in question.


Regards, Peter


On 06/13/2016 07:18 PM, Peter Levart wrote:

Hi,

I explored various strategies to minimize worst-case lookup 
performance for MethodType keys in LinearProbeHashtable. One idea is 
from the "Hopscotch hashing" algorithm [1] which tries to optimize 
placement of keys by moving them around at each insertion or deletion. 
While a concurrent Hopscotch hashtable is possible, it requires 
additional "metadata" about buckets which complicates it and does not 
make it practical for implementing in Java until Java 

RFR: JDK-8031043 ClassValue's backing map should have a smaller initial size

[Peter Levart]

Hi,

The patch for this issue is a result of the following discussion on 
mlvm-dev list:


http://mail.openjdk.java.net/pipermail/mlvm-dev/2016-May/006602.html

I propose the following improvement to the implementation of ClassValue API:

http://cr.openjdk.java.net/~plevart/misc/ClassValue.Alternative2/webrev.04.3/

This is a re-implementation of ClassValue API that leverages a 
ConcurrentMap for storage. ConcurrentHashMap was initially tested, but 
using it regressed lookup performance for about ~10%. So a simple 
ConcurrentMap implementation (LinearProbeHashtable) was created which 
has smaller footprint, better CPU cache locality and consequently faster 
lookup performance for the task at hand. The goal of this task was to 
shrink the memory footprint of ClassValue API. To see the end effect 
let's first look at the results of a benchmark that simulates a 
re-deployment of an application in an app server where several apps are 
deployed and each uses it's own set of classes and ClassValue(s):


http://cr.openjdk.java.net/~plevart/misc/ClassValue.Alternative2/ClassValueExpungeBench.java

Benchmark (classValuesPerPart) 
(classesPerPart)  (impl)  (partitions)  Mode  Cnt Score Error  Units
ClassValueExpungeBench.redeployPartition 8  1024
jdk916ss   1665.682 ± 1.485  ms/op
ClassValueExpungeBench.redeployPartition 8  4096
jdk916ss   16   247.040 ± 7.684  ms/op
ClassValueExpungeBench.redeployPartition 64  1024
jdk916ss   16   302.536 ± 27.750  ms/op
ClassValueExpungeBench.redeployPartition 64  4096
jdk916ss   16  1174.002 ± 77.183  ms/op


Benchmark (classValuesPerPart)  (classesPerPart)  (impl)  (partitions) 
Mode  CntScore   Error  Units
ClassValueExpungeBench.redeployPartition 8  1024  
pl04.316ss   16   47.179 ± 1.436  ms/op
ClassValueExpungeBench.redeployPartition 8  4096  
pl04.316ss   16  163.067 ± 8.118  ms/op
ClassValueExpungeBench.redeployPartition 64  1024  
pl04.316ss   16   67.581 ± 1.718  ms/op
ClassValueExpungeBench.redeployPartition 64  4096  
pl04.316ss   16  240.458 ± 6.616  ms/op


A by-product of this simulation is a heap dump histogram taken after the 
last warmup iteration when all "applications" are deployed:


top-10 classes heap dump for 64 ClassValue(s) x 4096 Class(es) x 16 
partitions


jdk9:

 num #instances #bytes  class name (module)
---
   1:   4194309  167772360 java.util.WeakHashMap$Entry 
(java.base@9-internal)
   2:   4195329  134250528 java.lang.ClassValue$Entry 
(java.base@9-internal)
   3: 65539   34603248 [Ljava.util.WeakHashMap$Entry; 
(java.base@9-internal)
   4: 65537   34603032 [Ljava.lang.ClassValue$Entry; 
(java.base@9-internal)

   5: 673017552448  java.lang.Class (java.base@9-internal)
   6: 655364194304 java.lang.ClassValue$ClassValueMap 
(java.base@9-internal)
   7: 655522097664 java.lang.ref.ReferenceQueue 
(java.base@9-internal)
   8: 677361676344  [Ljava.lang.Object; 
(java.base@9-internal)

   9: 663491116848  [I (java.base@9-internal)
  10: 655541048864 java.lang.ref.ReferenceQueue$Lock 
(java.base@9-internal)

---
 388915640 == 370 MiB

pl04.3:

 num #instances #bytes  class name (module)
---
   1:133274   69833848  [Ljava.lang.Object; 
(java.base@9-internal)

   2: 673007552376  java.lang.Class (java.base@9-internal)
   3: 655362097152 java.lang.ClassValue$ClassValueMap 
(java.base@9-internal)
   4: 655362097152 
java.lang.ClassValue$ClassValueMap$WeakNode (java.base@9-internal)

   5: 663491116848  [I (java.base@9-internal)
   6: 659911055856  java.lang.Object (java.base@9-internal)
   7:  7434 696584  [B (java.base@9-internal)
   8:  1551 299680  [Ljava.lang.Class; 
(java.base@9-internal)
   9:  2447 176184  java.lang.reflect.Field 
(java.base@9-internal)

  10:  7154 171696  java.lang.String (java.base@9-internal)
---
  85097376 == 81 MiB


Footprint is reduced for more than 4 times. The main improvement of this 
implementation is that to support M Class(es) x N ClassValue(s), the 
only part of the data structure that has O(M*N) space footprint are the 
backing array(s) of LinearProbeHashtable. No less than 3 and no more 
than 6 Object slots are used per Class x ClassValue association (4 in 
average). Other 

Re: proxy an interface and call a default method

[Peter Levart]

On 06/02/2016 06:34 PM, fo...@univ-mlv.fr wrote:


So perhaps, instead of providing a Proxy::findSuper method that
returns a pre-bound MH, there could simply be a method like the
following in the Proxy class:

public final Object invokeSuper(Class interfaze, String
methodName, MethodType methodType, Object ... args) { ... }

What do you think?


yes, good idea,
i think it should be static (and takes a Proxy as parameter) to avoid 
unwanted overriding.


Something like the following?

http://cr.openjdk.java.net/~plevart/jdk9-dev/Proxy.invokeSuperDefaults/webrev.02/

Usage is even simpler with this API:

public class Test {

interface I1 {
default void m() {
System.out.println("default I1.m() called");
}
}

interface I2 {
default void m() {
System.out.println("default I2.m() called");
}
}

interface I12 extends I1, I2 {
@Override
void m();
}

public static void main(String[] args) {

InvocationHandler h = (proxy, method, params) -> {
System.out.println("InvocationHandler called for: " + method);
try {
return Proxy.invokeSuper(proxy, method, params);
} catch (InvocationTargetException e) {
throw e.getCause();
}
};

I1 i1 = (I1) Proxy.newProxyInstance(
I1.class.getClassLoader(), new Class[]{I1.class}, h);
i1.m();

I2 i2 = (I2) Proxy.newProxyInstance(
I2.class.getClassLoader(), new Class[]{I2.class}, h);
i2.m();

I12 i12 = (I12) Proxy.newProxyInstance(
I12.class.getClassLoader(), new Class[]{I12.class}, h);
i12.m();
}
}


Gives the following output:

InvocationHandler called for: public default void Test$I1.m()
default I1.m() called
InvocationHandler called for: public default void Test$I2.m()
default I2.m() called
InvocationHandler called for: public abstract void Test$I12.m()
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at $Proxy2.m(Unknown Source)
at Test.main(Test.java:49)
Caused by: java.lang.IllegalAccessException: no such method: 
Test$I12.m()void/invokeSpecial
at 
java.lang.invoke.MemberName.makeAccessException(java.base@9-internal/MemberName.java:928)
at 
java.lang.invoke.MemberName$Factory.resolveOrFail(java.base@9-internal/MemberName.java:1064)
at 
java.lang.invoke.MethodHandles$Lookup.resolveOrFail(java.base@9-internal/MethodHandles.java:1692)
at 
java.lang.invoke.MethodHandles$Lookup.findSpecial(java.base@9-internal/MethodHandles.java:1150)
at 
java.lang.reflect.Proxy.invokeSuper(java.base@9-internal/Proxy.java:1151)

at Test.lambda$main$0(Test.java:33)
... 2 more
Caused by: java.lang.AbstractMethodError: Test$I12.m()V
at 
java.lang.invoke.MethodHandleNatives.resolve(java.base@9-internal/Native 
Method)
at 
java.lang.invoke.MemberName$Factory.resolve(java.base@9-internal/MemberName.java:1036)
at 
java.lang.invoke.MemberName$Factory.resolveOrFail(java.base@9-internal/MemberName.java:1061)

... 6 more


...which is expected. You can't call the super abstract method. You have 
to resolve the Method object of a particular interface (I1 or I2) 
yourself in such case.


I think this is a simple API that everyone could understand.

Regards, Peter

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: proxy an interface and call a default method

[Peter Levart]

Hi Remi, Jochen,


On 06/02/2016 11:15 AM, fo...@univ-mlv.fr wrote:

>The solution could be for Proxy API to provide a MH that was already
>bound to the Proxy instance. Such pre-bound MH could not be abused then.

independently of any security issue, it may be a good idea but doing a partial 
evaluation on a MH is not cheap.



I created a prototype for this:

http://cr.openjdk.java.net/~plevart/jdk9-dev/Proxy.invokeSuperDefaults/webrev.01/

Example usage is as follows:

public class Test {

interface I {
default void m() {
System.out.println("default I.m() called");
}
}

public static void main(String[] args) {

InvocationHandler h = (proxy, method, params) -> {
System.out.println("InvocationHandler called for: " + method);
MethodHandle superM = ((Proxy) proxy).findSuper(I.class, 
"m", MethodType.methodType(void.class));

return superM.invokeWithArguments(params);
};

I i = (I) Proxy.newProxyInstance(
I.class.getClassLoader(), new Class[]{I.class}, h);

i.m();
}
}


It works, but in order for this to have adequate performance, caching 
would have to be added. But caching a pre-bound MH would require caching 
on per-proxy-instance basis, which would not be very efficient. So 
perhaps, instead of providing a Proxy::findSuper method that returns a 
pre-bound MH, there could simply be a method like the following in the 
Proxy class:


public final Object invokeSuper(Class interfaze, String methodName, 
MethodType methodType, Object ... args) { ... }


What do you think?

Regards, Peter

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: proxy an interface and call a default method

[Peter Levart]

On 05/27/2016 01:40 PM, Remi Forax wrote:

I don't see the issue if the lookup object represent the proxy class itself 
restricted to only access to public methods.

Rémi


I think there could be security issues. For example, let there be an 
interface like:


public interface Resource {
InputStream openStream() throws IOException;

default byte[] getContent() throws IOException {
try (InputStream is = openStream()) {
return is.readAllBytes();
}
}
}

Then there might be an implementation that overrides both methods, like:

public class SecuredResource implements Resource {

@CallerSensitive
@Override
public InputStream openStream() throws IOException {
checkPermission(Reflection.getCallerClass());
return openStreamImpl();
}

@CallerSensitive
@Override
public byte[] getContent() throws IOException {
checkPermission(Reflection.getCallerClass());
try (InputStream is = openStreamImpl()) {
return is.readAllBytes();
}
}

private InputStream openStreamImpl() throws IOException { ...
...
}


If Proxy API allowed access to a MH(invokespecial Resource::getContent), 
then one could abuse such MH to circumvent access check in an instance 
of SecuredResource by making it appear it was invoked from the Resource 
class.


I think that bytecode verifier does not allow such things and 
SecuredResource can be considered perfectly safe in this respect.


The solution could be for Proxy API to provide a MH that was already 
bound to the Proxy instance. Such pre-bound MH could not be abused then.


Regards, Peter



- Mail original -

De: "Peter Levart" <peter.lev...@gmail.com>
À: "Da Vinci Machine Project" <mlvm-dev@openjdk.java.net>, "jochen Theodorou" 
<blackd...@gmx.org>
Envoyé: Vendredi 27 Mai 2016 12:50:34
Objet: Re: proxy an interface and call a default method

Hi,

I think the main problem here is that by providing the InvocationHandler
with a Lookup that could provide "invokespecial" MHs for the proxy
interface(s) could be abused. Anyone can create a Proxy for any public
interface and supply its own InvocationHandler which could be used to
"steal" such Lookup object.

There would have to be a way to restrict calling interface "super"
methods from InvocationHandler *INSTANCES* that are bound to particular
Proxy instances.

Hm...

Regards, Peter

On 05/26/2016 08:20 AM, Jochen Theodorou wrote:

Hi all,

I am looking for a solution to the following problem... I have an
interface and an object that is supposed to serve as implementation,
but does not implement the interface. n methods of the interface will
be redirected to the object, but in case of default methods I would
like to have the implementation provided by the interface. I am
looking especially for a solution without me generating classes at
runtime by hand.

Now there are several problems... I seem not to be able to invoke a
default method by reflection. By MethodHandles I did something like this:


MethodHandles.Lookup.class.getDeclaredConstructor(Class.class,
int.class).
   newInstance(interfaceClass, MethodHandles.Lookup.PRIVATE).
   unreflectSpecial(method, interfaceClass).
   bindTo(receiver);

where receiver is a dynamic proxy, method the Method of the default
method, interfaceClass the Class of the interface with the default
method.

But I am calling a private constructor here, which is bad, plus the
above procedure does no longer work on JDK9.

So what am I supposed to do? change from a proxy to runtime generated
classes and hope the best for classloaders and modules not getting in
my way?

bye Jochen
___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev



___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: ClassValue perf?

[Peter Levart]

On 05/30/2016 11:09 PM, Peter Levart wrote:

Will revert that to volatile semantics...


So, here it is. The latest incarnation - the same as webrev.04.1 but 
with volatile semantics:


http://cr.openjdk.java.net/~plevart/misc/ClassValue.Alternative2/webrev.04.2/

Benchmarks show no change in performance:

http://cr.openjdk.java.net/~plevart/misc/ClassValue.Alternative2/webrev.04.2.bench_results.txt


Regards, Peter

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: ClassValue perf?

[Peter Levart]

On 05/30/2016 07:47 PM, Aleksey Shipilev wrote:

On 05/30/2016 06:59 PM, Peter Levart wrote:

I also employed get-acquire/put-release memory ordering semantics
instead of SC (volatile) in hope that it might improve a bit the
performance on platforms such as PowerPC or ARM, but this can be changed
back to SC if anyone gets scared of it :-)

Revert, you're playing with fire here. Your _default_ modus operandi
should be "scared" when dealing with concurrency. The correctness with
acq/rel should be proven separately, and not by empirical testing. There
is a reason why putOrdered is not used everywhere.

Thanks,
-Aleksey


Hi Aleksey,

You are right.  Users kind of expect from a general construct like 
ConcurrentMap to behave in a SC manner. Although my use of 
get-acquire/put-release in a Map-like construct could be correct by 
itself, it might be surprising for users to observe things like:


final static LinearProbeHashtable<Integer, Integer> t = new 
LinearProbeHashtable<>();


Thread A:
t.put(1, 10);

Thread B:
t.put(2, t.get(1));

Thread C:
Integer v2 = t.get(2);
Integer v1 = t.get(1);

with a possible outcome of (v1, v2) being (null, 10);

Will revert that to volatile semantics...

Regards, Peter





___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: ClassValue perf?

[Peter Levart]

ful piece of work.

Am 26.05.2016 um 10:59 schrieb Peter Levart <peter.lev...@gmail.com 
<mailto:peter.lev...@gmail.com>>:

How does this implementation compare on your hardware, Michael?


Results attached. It improves on the unpatched version in all cases, 
is in most cases even faster than the "simple solution" (reduce 
initial size to 1), and reduces complexity of ClassValue. It passes 
all open and closed jli-related tests as well as the Nashorn tests. 
Looking really good.


Let me run the full internal test suite across platforms.

Best,

Michael



--

Oracle <http://www.oracle.com/>
Dr. Michael Haupt | Principal Member of Technical Staff
Phone: +49 331 200 7277 | Fax: +49 331 200 7561
OracleJava Platform Group | LangTools Team | Nashorn
Oracle Deutschland B.V. & Co. KG | Schiffbauergasse 14 | 14467 
Potsdam, Germany


ORACLE Deutschland B.V. & Co. KG | Hauptverwaltung: Riesstraße 25, 
D-80992 München

Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V. | Hertogswetering 
163/167, 3543 AS Utrecht, Niederlande

Handelsregister der Handelskammer Midden-Nederland, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher
Green Oracle <http://www.oracle.com/commitment> 	Oracle is committed 
to developing practices and products that help protect the environment





___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: proxy an interface and call a default method

[Peter Levart]

Hi,

I think the main problem here is that by providing the InvocationHandler 
with a Lookup that could provide "invokespecial" MHs for the proxy 
interface(s) could be abused. Anyone can create a Proxy for any public 
interface and supply its own InvocationHandler which could be used to 
"steal" such Lookup object.


There would have to be a way to restrict calling interface "super" 
methods from InvocationHandler *INSTANCES* that are bound to particular 
Proxy instances.


Hm...

Regards, Peter

On 05/26/2016 08:20 AM, Jochen Theodorou wrote:

Hi all,

I am looking for a solution to the following problem... I have an 
interface and an object that is supposed to serve as implementation, 
but does not implement the interface. n methods of the interface will 
be redirected to the object, but in case of default methods I would 
like to have the implementation provided by the interface. I am 
looking especially for a solution without me generating classes at 
runtime by hand.


Now there are several problems... I seem not to be able to invoke a 
default method by reflection. By MethodHandles I did something like this:


MethodHandles.Lookup.class.getDeclaredConstructor(Class.class, 
int.class).

  newInstance(interfaceClass, MethodHandles.Lookup.PRIVATE).
  unreflectSpecial(method, interfaceClass).
  bindTo(receiver);


where receiver is a dynamic proxy, method the Method of the default 
method, interfaceClass the Class of the interface with the default 
method.


But I am calling a private constructor here, which is bad, plus the 
above procedure does no longer work on JDK9.


So what am I supposed to do? change from a proxy to runtime generated 
classes and hope the best for classloaders and modules not getting in 
my way?


bye Jochen
___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: ClassValue perf?

[Peter Levart]

Hi Michael,


On 05/23/2016 03:56 PM, Michael Haupt wrote:
I've ran the unpatched version and Peter's two patches once more. The 
results are attached (results.txt). They confirm Aleksey's observation.


Regarding the 03 patch (plevart3 column in the results), perfasm 
output (see http://cr.openjdk.java.net/~mhaupt/8031043/perfasm.zip 
) suggests 
the cost is mainly accrued in ConcurrentHashMap. The same is the case 
for the 02 patch (plevart2 column).


As things stand, I think we can even focus on Peter's 02 patch, as 
this is the faster of his two proposals (plevart2 column in the 
results), reduces the footprint, and reduces the implementation 
complexity. Can anything be done to improve on its performance? (It 
has slight performance slowdowns for the single-value case as well.)


I can't think of anything else short of improving performance of CHM itself.

Or replacing CHM with a "better" implementation:

http://cr.openjdk.java.net/~plevart/misc/ClassValue.Alternative2/webrev.04/

This webrev is similar to webrev.02. It's only difference is in 
ClassValueMap which extends LinearProbeHashtable instead of 
ConcurrentHashMap. LinearProbeHashtable is a simple implementation of a 
linear-probe hash table. It's not a full Map implementation. It only 
implements methods needed in ClassValue. With this implementation I get 
a slight boost compared to JDK 9 ClassValue implementation for all sizes 
and counts:


Benchmark (classCount)  (classValueCount) 
(impl)  Mode  Cnt   Score   Error  Units
ClassValueBench.randomAccess   128  1 jdk9  
avgt   10   9.079 ± 0.092  ns/op
ClassValueBench.randomAccess   128  4 jdk9  
avgt   10  10.615 ± 0.102  ns/op
ClassValueBench.randomAccess   128 16 jdk9  
avgt   10  11.665 ± 0.012  ns/op
ClassValueBench.randomAccess   128256 jdk9  
avgt   10  19.151 ± 0.219  ns/op
ClassValueBench.randomAccess  1024  1 jdk9  
avgt   10  14.642 ± 0.425  ns/op
ClassValueBench.randomAccess  1024  4 jdk9  
avgt   10  22.577 ± 0.093  ns/op
ClassValueBench.randomAccess  1024 16 jdk9  
avgt   10  19.864 ± 0.736  ns/op
ClassValueBench.randomAccess  1024256 jdk9  
avgt   10  60.470 ± 0.285  ns/op
ClassValueBench.sequentialAccess   128  1 jdk9  
avgt   10   9.741 ± 0.033  ns/op
ClassValueBench.sequentialAccess   128  4 jdk9  
avgt   10   8.252 ± 0.029  ns/op
ClassValueBench.sequentialAccess   128 16 jdk9  
avgt   10   7.888 ± 1.249  ns/op
ClassValueBench.sequentialAccess   128256 jdk9  
avgt   10  16.493 ± 0.415  ns/op
ClassValueBench.sequentialAccess  1024  1 jdk9  
avgt   10  13.376 ± 0.452  ns/op
ClassValueBench.sequentialAccess  1024  4 jdk9  
avgt   10  10.023 ± 0.020  ns/op
ClassValueBench.sequentialAccess  1024 16 jdk9  
avgt   10   8.029 ± 0.178  ns/op
ClassValueBench.sequentialAccess  1024256 jdk9  
avgt   10  33.472 ± 0.058  ns/op


Benchmark (classCount)  (classValueCount) 
(impl)  Mode  Cnt   Score   Error  Units
ClassValueBench.randomAccess   128  1 pl04  
avgt   10   8.955 ± 0.055  ns/op
ClassValueBench.randomAccess   128  4 pl04  
avgt   10   9.999 ± 0.017  ns/op
ClassValueBench.randomAccess   128 16 pl04  
avgt   10  11.615 ± 1.928  ns/op
ClassValueBench.randomAccess   128256 pl04  
avgt   10  17.063 ± 0.460  ns/op
ClassValueBench.randomAccess  1024  1 pl04  
avgt   10  12.553 ± 0.086  ns/op
ClassValueBench.randomAccess  1024  4 pl04  
avgt   10  16.766 ± 0.221  ns/op
ClassValueBench.randomAccess  1024 16 pl04  
avgt   10  18.496 ± 0.051  ns/op
ClassValueBench.randomAccess  1024256 pl04  
avgt   10  41.390 ± 0.321  ns/op
ClassValueBench.sequentialAccess   128  1 pl04  
avgt   10   7.854 ± 0.381  ns/op
ClassValueBench.sequentialAccess   128  4 pl04  
avgt   10   7.498 ± 0.055  ns/op
ClassValueBench.sequentialAccess   128 16 pl04  
avgt   10   9.218 ± 1.000  ns/op
ClassValueBench.sequentialAccess   128256 pl04  
avgt   10  13.593 ± 0.275  ns/op
ClassValueBench.sequentialAccess  1024  1 pl04  
avgt   10   8.774 ± 0.037  ns/op
ClassValueBench.sequentialAccess  1024  4 pl04  
avgt   10   8.562 ± 0.014  ns/op
ClassValueBench.sequentialAccess  1024 16 pl04  
avgt   10   7.596 ± 0.027  ns/op

Re: Question in understanding ClassValue better

[Peter Levart]

On 05/24/2016 01:41 PM, Peter Levart wrote:

Hm,

It seems that my example will not help much. It also seems that the 
only problem with plain:


static final ClassValue META_CLASS_CV = new 
ClassValue() {

@Override
 protected MetaClass computeValue(Class type) {
 return new MetaClass(type);
 }
};

...is the fact that MetaClass is a class loaded by non-bootstrap class 
loader and that in case this is a Web app class loader, it prevents 
undeployment. Can you confirm that a MetaClass instance only 
references the 'type' Class it is derived from (it's Methods, 
Fields, etc.) and never references objects from any child class 
loaders of the type's class loader?


If that is the case, then you could replace MetaClass with a generic 
data structure, composed of instances of bootstrap classes (HashMap, 
ArrayList, Object[], ...). That way, Groovy runtime class loader will 
not be "captured" by a reference from an aClass loaded by bootstrap 
class loader.


Is MetaClass a complicated data structure?


...peeking at Groovy sources, very much so. There's a solution though. 
Various Meta* classes in Groovy runtime reference at some point the 
reflective objects (Class, Method, Constructor, Field) describing the 
'type' they are derived from.


Every reference to a Class object from such Meta* object should be 
wrapped in something like the following:



public final class ClassReference extends WeakReference<Class>
implements Supplier<Class> {

private static final ConcurrentHashMap<ClassReference, 
ClassReference> MAP

= new ConcurrentHashMap<>();
private static final ReferenceQueue<Class> QUEUE
= new ReferenceQueue<>();

public static ClassReference forClass(Class clazz) {
ClassReference oldRef;
while ((oldRef = (ClassReference) QUEUE.poll()) != null) {
MAP.remove(oldRef);
}
ClassReference newRef = new ClassReference(clazz);
oldRef = MAP.putIfAbsent(newRef, newRef);
return oldRef == null ? newRef : oldRef;
}

private final String name;
private final int hash;

private ClassReference(Class clazz) {
super(clazz, QUEUE);
name = clazz.getName();
hash = clazz.hashCode();
}

@Override
public Class get() {
Class clazz = super.get();
if (clazz == null) {
throw new IllegalStateException(
"Class " + name + " has already been unloaded");
}
return clazz;
}

@Override
public int hashCode() {
return hash;
}

@Override
public boolean equals(Object obj) {
Class clazz;
return obj == this ||
   (obj instanceof ClassReference &&
(clazz = get()) != null &&
clazz == ((ClassReference) obj).get());
}
}


Every reference to a Method should be wrapped in something like this:


public final class MethodReference implements Supplier {
private static final ClassValue<Method[]> DECLARED_METHODS_CV =
new ClassValue<Method[]>() {
@Override
protected Method[] computeValue(Class type) {
return type.getDeclaredMethods();
}
};

private final ClassReference declaringClassRef;
private final int index;

public MethodReference(Method method) {
Class declaringClass = method.getDeclaringClass();
declaringClassRef = ClassReference.forClass(declaringClass);
Method[] methods = DECLARED_METHODS_CV.get(declaringClass);
index = Arrays.asList(methods).indexOf(method);
}

@Override
public Method get() {
return DECLARED_METHODS_CV.get(declaringClassRef.get())[index];
}

@Override
public int hashCode() {
return declaringClassRef.hashCode() * 31 + index;
}

@Override
public boolean equals(Object obj) {
return obj == this || (
obj instanceof MethodReference &&
((MethodReference) obj).declaringClassRef == 
this.declaringClassRef &&

((MethodReference) obj).index == this.index
);
}
}


And similar with every reference to a Constructor or Field.


In addition, the MetaClass structure should be isolated from the class 
it is derived from with what I presented in the previous message 
(ClassValue<WeakReference> + ArrayList referenced 
from MetaClass static field)


Would that work?



Regards, Peter


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: Question in understanding ClassValue better

[Peter Levart]

On 05/24/2016 10:26 AM, Jochen Theodorou wrote:
Peter, I fully understand if you cannot reply to this mail easily, 
just wanted to ping to ensure this is not forgotten ;)


Sorry Jochen, I forgot about it... Thanks for reminding me!



On 20.05.2016 01:33, Jochen Theodorou wrote:

On 19.05.2016 21:32, Peter Levart wrote:
[...]

a ClassValue instance can be thought of as a component of a compound
key. Together with a Class, they form a tuple (aClass, aClassValue) 
that

can be associated with an "associated value", AV. And yes, the AVs
associated with tuple containing a particular Class are strongly
reachable from that Class.


I see, I was mixing ClassValue and AV, I was more talking about AV, than
ClassValue itself, though ClassValue surely plays an important role in
here. Anyway, I am going for that AV is the value computed by 
aClassValue.


You said that the AV is strongly reachable from aClass. Can I further
assume, that aClassValue is not strongly reachable from aClass? And that
aClassValue can be collected independent of aClass? Can I further
assume, that aClassValue can be collected even if AVs for it continue to
exist?


ClassValue implementation is such that aClassValue is not strongly 
reachable from aClass as a consequence of (aClassValue, aClass) -> AV 
association. It can be reachable because of other unrelated references. 
Therefore, the answer is YES for all above questions.




[...]

An AV associated with a tuple (Integer.TYPE, aClassValue) -> AV can be
garbage collected. But only if aClassValue can be garbage collected 
1st.


hmm... so in (aClass, aClassValue)->AV if aClassValue can be collected,
AV can, but not the other way around...


It is a two stage process with a GC cycle between the stages. In 1st 
stage aClassValue is unreferenced, then GC kicks-in, collects 
aClassValue and enqueues a WeakReference that was pointing to 
aClassValue, then the enqueued WeakReference triggers expunging of 
associated AV on next access to some association (???, aClass) for the 
same aClass. That's how it is implemented in current ClassValue. If 
there's no "next access", then there's no expunging and the AV remains 
reachable.



what about aClass? if nothing
but AV is referencing aClass, can AV be garbage collected, even if
aClassValue cannot? Can I extend your statement to AV can be collected
only if either aClass or aClassValue can be garbage collected first?


aClass is different from aClassValue in that aClass and all associated 
AVs can be collected at the same time. AVs are reachable from associated 
aClass, but if aClass is not strongly reachable from anywhere, they can 
all be collected together. aClassValue reachability does not play a role 
here.




Let us assume this is the case for now.


This is the most tricky part to get right in order to prevent leaks. If
in above example, aClassValue is reachable from the AV, then we have a
leak. The reachability of a ClassValue instance from the associated
value AV is not always obvious. One has to take into account the
following non-obvious references:

1 - each object instance has an implicit reference to its implementing
class
2 - each class has a reference to its defining ClassLoader
3 - each ClassLoader has a reference to all classes defined by it
(except VM annonymous classes)
4 - each ClassLoader has a reference to all its predecessors (that it
delegates to)

Since a ClassValue instance is typically assigned to a static final
field, such instance is reachable from the class that declares the 
field.


I think you can get the picture from that.


yeah... that is actually problematic. Because if I keep no hard
reference the ClassValue can be collected, even if the AVs still
exist... 


you need aClassValue to lookup the AVs. Without aClassValue, they are 
not retrievable. So you better keep a reference to aClassValue as long 
as you need it to lookup the AVs...



meaning they would become unreachable. And if I keep one I have
a memory leak... well more about in the program you have shown me 
later on.


[...]

Ok, let's set up the stage. If I understand you correctly, then:

Groovy runtime is loaded by whatever class loader is loading the
application (see the comment in MetaClass constructor if this is not
true).  This is either the ClassLoader.getSystemClassLoader() (the APP
class loader) if started from command line or for example Web App class
loader in a Web container.


Well, actually... if you start a script on the command line, the loader
is a child to the app class loader, when used as library it could be the
app loader (for example if the groovy program is precompiled) and in a
tomcat like scenario it could be either the class loader for the web
app, or the loader for all web apps. But let's go with the cases you
mentioned first ;)

MetaClass(es) are objects implemented by Groovy runtime class(es). 
Let's

call them simply MetaClass.


good


Here's how I would do that:


public class MetaCla

Re: Question in understanding ClassValue better

[Peter Levart]

A small correction...


On 05/19/2016 09:32 PM, Peter Levart wrote:

// the public API
public MetaClass getInstanceFor(Class type) {


This should of course read:

 // the public API
public *static* MetaClass getInstanceFor(Class type) {


Regards, Peter

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: Question in understanding ClassValue better

[Peter Levart]

Hi Jochen,

I'll try to answer your questions as profoundly as I can...


On 05/19/2016 04:27 PM, Jochen Theodorou wrote:

Hi,

at the beginning of this year I had an exchange with Peter Lavart 
about JDK-8136353 (see 
http://mail.openjdk.java.net/pipermail/mlvm-dev/2016-January/006563.html), 
and that is probably based on wrong assumptions. But I must confess I 
still have trouble understanding ClassValue semantics. And since the 
ClassValue problem in Groovy came up again, I though I make another 
try based on a list of assumptions and asking if they are wrong or right


1) ClassValue can be basically understood as a strong reference of a 
class to a class value


a ClassValue instance can be thought of as a component of a compound 
key. Together with a Class, they form a tuple (aClass, aClassValue) that 
can be associated with an "associated value", AV. And yes, the AVs 
associated with tuple containing a particular Class are strongly 
reachable from that Class.


2) a ClassValue associated with a system class (for example 
Integer.TYPE) is never garbage collected


An AV associated with a tuple (Integer.TYPE, aClassValue) -> AV can be 
garbage collected. But only if aClassValue can be garbage collected 1st. 
This is the most tricky part to get right in order to prevent leaks. If 
in above example, aClassValue is reachable from the AV, then we have a 
leak. The reachability of a ClassValue instance from the associated 
value AV is not always obvious. One has to take into account the 
following non-obvious references:


1 - each object instance has an implicit reference to its implementing class
2 - each class has a reference to its defining ClassLoader
3 - each ClassLoader has a reference to all classes defined by it 
(except VM annonymous classes)
4 - each ClassLoader has a reference to all its predecessors (that it 
delegates to)


Since a ClassValue instance is typically assigned to a static final 
field, such instance is reachable from the class that declares the field.


I think you can get the picture from that.

3) a ClassValue from a different loader than the system loader, 
associated with a system class, will prevent that loader to unload


You mean an instance of a ClassValue subclass loaded by a non-system 
class loader? I don't think this matters much. Any object instance with 
a runtime class that is not a system class, while being reachable, holds 
the non-system ClassLoader non-reclaimable (this follows from the 1st 
and 2nd rules of non-obvious references above)


a ClassValue instance is not associated with a Class. (aClass, 
aClassValue) tuple is associated with an associated value AV. Such 
association is implemented in a way where aClassValue is not strongly 
reachable from aClass BECAUSE OF THE ASSOCIATION ITSELF, if that is what 
you wanted to know.


4) a ClassValue referencing to the class it is associated with, does 
not prevent the collection of that class


An association (aClass, aClassValue) -> AV is implemented in a way where 
aClass is not strongly reachable from aClassValue BECAUSE OF THE 
ASSOCIATION ITSELF. It can still be reachable because of other 
non-obvious references mentioned above.




Point 2 and 3 are kind of problematic for me and I wish them wrong, 
but they would follow from 1. The exchange with Peter makes me think 
assumption 4 is wrong... just I don't understand why.


If those assumptions are right, then I actually wonder in what cases I 
should use ClassValue without causing memory leaks. What I wanted to 
use it for is to associate a meta class with every class I need a meta 
class for. This includes system classes. If 3 is right, then doing so 
would prevent the Groovy runtime from being unloaded. Even if the meta 
classes are able to unload, the implementation of the ClassValue would 
still be there. And since that comes from the same loader, that loaded 
the runtime, that loader will stay. Now loading and (trying to) unload 
the Groovy runtime countless times would end up in a OOME at some 
point (permgen problem in older JDKs). And even if I would do 
something else for class from the standard loaders, I would still get 
into trouble on for example Tomcat. Not to forget that having two 
parallel structures for this raises the question as of why to use 
ClassValue at all.


I think what it boils down to in the end is: When (under what 
conditions) for what to use ClassValue at all.


bye Jochen



Ok, let's set up the stage. If I understand you correctly, then:

Groovy runtime is loaded by whatever class loader is loading the 
application (see the comment in MetaClass constructor if this is not 
true). This is either the ClassLoader.getSystemClassLoader() (the APP 
class loader) if started from command line or for example Web App class 
loader in a Web container.


MetaClass(es) are objects implemented by Groovy runtime class(es). Let's 
call them simply MetaClass.



Here's how I would do that:


public class MetaClass {

// this list keeps 

Re: ClassValue perf?

[Peter Levart]

Hi Michael,


On 05/06/2016 04:48 PM, Michael Haupt wrote:

Hi Peter,

thank you. I've run the full benchmark in my setup and uploaded the 
updated cumulative results to 
http://cr.openjdk.java.net/~mhaupt/8031043/ 
<http://cr.openjdk.java.net/%7Emhaupt/8031043/>.


The benchmark indeed shows that this latest addition to the group 
slows down random and sequential access, especially for small numbers 
of values and classes. The OpenJDK tests are fine; I'm running a batch 
of internal tests as well.


Given that one concern with this issue, next to reducing footprint, 
was to optimise for the single-value case, I'm still a bit hesitant 
even though the sheer amount of code reduction is impressive. I'll 
evaluate further.


Interesting. I observed quite the opposite on my machine (i7-4771, 8 MiB 
cache) . For sequential access pattern or for random access with small 
number of CV(s) and Class(es) the results are comparable. Only for 256 
CV(s) x 1024 Class(es) and with random access pattern, I observed about 
20% drop of performance which I attributed to the difference in design 
of CHM vs. the 'cache' of JDK 9 ClassValue (worse CPU cache locality for 
CHM):


http://cr.openjdk.java.net/~plevart/misc/ClassValue.Alternative2/ClassValueBench.java

I doubt that single value per Class instance is something that is 
beneficial to optimize. Such optimization would be very fragile so it 
would not be something to rely on. Typical or even worst case 
performance is more important in my opinion.


The fast-path lookup performance is the most important performance 
aspect of ClassValue, but it is not the only one that can be observed. 
Footprint and consequential GC / expunging overhead is also something to 
consider. The implementation presented in my webrev.02 maintains a 
linked list of weakly-referenced ClassValueMap(s). For each stale 
dequeued key, it probes each map and removes such key from any live 
map(s) containing it. This works optimally when the matrix of 
(ClassValue, Class) pairs is not sparse. I did an experiment with 
alternative expunging design where I maintain an array of 
weakly-referenced ClassValueMap(s) on each key that is inserted in them. 
This has approx. 10% additiona footprint overhead compared to original 
expunging design (but still just half the footprint overhead of jdk 9 
ClassValue design):


http://cr.openjdk.java.net/~plevart/misc/ClassValue.Alternative2/webrev.03/

The situation I envisioned was when a single JVM hosts multiple (say N) 
isolated applications (in an app server for example) and when one such 
application is re-deployed.


In original design (webrev.02) each dequeued ClassValue.key is probed 
against all class maps that remain and belong to the other N-1 
applications. In the alternative expunging design (webrev.03) the 
dequeued key just scans the array of weakly-referenced maps that the key 
was inserted into.


I created a benchmark to exercise such situation(s):

http://cr.openjdk.java.net/~plevart/misc/ClassValue.Alternative2/ClassValueExpungeBench.java

It measures the time of a hypothetical redeployment of one application 
in an app server where there are 16 such running applications. The 
measurement includes class-loading, GC time and initialization of 
ClassValue(s). Results show that alternative expunging design 
(webrev.03) doesn't bring any improvements (or that original supposedly 
sub-optimal expunging design (webrev.02) doesn't show any weaknesses) 
for the range of parameters exercised in the benchmark.


What this benchmark shows too is that original jdk 9 ClassValue has at 
least 2x overhead with cleanup compared to my designs (note that 
benchmark includes classloading time too).


 Regards, Peter



Best,

Michael

Am 05.05.2016 um 17:21 schrieb Peter Levart <peter.lev...@gmail.com 
<mailto:peter.lev...@gmail.com>>:


Hi Michael,


On 05/04/2016 06:02 PM, Michael Haupt wrote:

Hi Peter,

thank you for chiming in again! :-) I'll look at this in depth on 
Friday.


Good. Because I found bugs in expunging logic and a discrepancy of 
behavior when a value is installed concurrently by some other thread 
and then later removed while the 1st thread is still calculating the 
value. Current ClassValue re-tries the computation until it can make 
sure there were no concurrent changes to the entry during its 
computation. I fixed both things and verified that the behavior is 
now the same:


http://cr.openjdk.java.net/~plevart/misc/ClassValue.Alternative2/webrev.02/

Regards, Peter



--

Oracle <http://www.oracle.com/>
Dr. Michael Haupt | Principal Member of Technical Staff
Phone: +49 331 200 7277 | Fax: +49 331 200 7561
OracleJava Platform Group | LangTools Team | Nashorn
Oracle Deutschland B.V. & Co. KG | Schiffbauergasse 14 | 14467 
Potsdam, Germany


ORACLE Deutschland B.V. & Co. KG | Hauptverwaltung: Riesstraße 25, 
D-80992 München

Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland V

Re: ClassValue perf?

[Peter Levart]

Hi Michael,


On 05/04/2016 06:02 PM, Michael Haupt wrote:

Hi Peter,

thank you for chiming in again! :-) I'll look at this in depth on Friday.


Good. Because I found bugs in expunging logic and a discrepancy of 
behavior when a value is installed concurrently by some other thread and 
then later removed while the 1st thread is still calculating the value. 
Current ClassValue re-tries the computation until it can make sure there 
were no concurrent changes to the entry during its computation. I fixed 
both things and verified that the behavior is now the same:


http://cr.openjdk.java.net/~plevart/misc/ClassValue.Alternative2/webrev.02/

Regards, Peter



Best,

Michael

Am 04.05.2016 um 17:50 schrieb Peter Levart <peter.lev...@gmail.com 
<mailto:peter.lev...@gmail.com>>:


Hi,


On 04/29/2016 10:28 AM, Michael Haupt wrote:

All,

see http://cr.openjdk.java.net/~mhaupt/8031043/ 
<http://cr.openjdk.java.net/%7Emhaupt/8031043/> for a snapshot of 
what is currently available.


We have three patches:
* Christian's, which simply reduces the HashMap size,
* Peter's, which refactors ClassValueMap into a WeakHashMap,
* mine, which attempts to introduce the single-value storage 
optimisation John had suggested (I worked on performance with 
Aleksey - thanks!).


All of these are collected in the patches subdirectory for 
convenience. (Peter, I adapted your patch to the new Unsafe location.)


I extended Peter's benchmark (thanks!) to cover single-value 
storage; the source code is in the benchmark subdirectory, together 
with raw results from running the benchmark with each of the three 
patches applied. A results-only overview is in benchmark-results.txt.


The three are roughly on par. I'm not sure the single-value storage 
optimisation improves much on footprint given the additional data 
that must be kept around to make transition to map storage safe.


Opinions?


I must admit that my old patch is very complex, so I doubt anyone 
will take time to review it. It is almost a clean-room 
re-implementation of ClassValue API. My main motivation was footprint 
optimization for all sizes - not just one value per class as I doubt 
this will be very common situation anyway. Current ClassValue 
maintains 2 parallel hash-tables per class. A WeakHashMap which is 
accessed with proper synchronization and an optimized "cache" of 
entries for quick access. This makes it consume almost 100 bytes per 
(Class, ClassValue) pair. I managed to almost half the overhead for 
typical situation (1024 classes x 16 ClassValue(s)), but for the 
price of complexity.


Reviving this thread made me think about ClassValue again and I got 
another idea. This is an experiment to see if ConcurrentHashMap could 
be leveraged to implement ClassValue API with little added complexity:


http://cr.openjdk.java.net/~plevart/misc/ClassValue.Alternative2/webrev.01/

And here are the results of a benchmark comparing JDK 9 original with 
this alternative:


http://cr.openjdk.java.net/~plevart/misc/ClassValue.Alternative2/ClassValueBench.java

It is a little slower for random access of bigger sizes and #s of 
classes. Most probably a consequence of reduced cache hit ratio as 
CHM is a classical hash table with buckets implemented as linked list 
of entries whereas jdk 9 ClassValue cache is a linear-scan hash table 
which has better cache locality. This is particularly obvious in 
sequential access where CHM behaves on-par. It's a pity that CHM has 
a non-changeable load factor of 0.75 as changing this to 0.5 would 
most certainly improve benchmark results for a little more memory.


Where this version excels is in footprint. I managed to more than 
half the overhead. There's only a single ReferenceQueue needed and 
consequently expunging of stale data is more prompt and thorough. The 
code of ClassValue has been more than halved too.


What do you think?

Regards, Peter



--

Oracle <http://www.oracle.com/>
Dr. Michael Haupt | Principal Member of Technical Staff
Phone: +49 331 200 7277 | Fax: +49 331 200 7561
OracleJava Platform Group | LangTools Team | Nashorn
Oracle Deutschland B.V. & Co. KG | Schiffbauergasse 14 | 14467 
Potsdam, Germany


ORACLE Deutschland B.V. & Co. KG | Hauptverwaltung: Riesstraße 25, 
D-80992 München

Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V. | Hertogswetering 
163/167, 3543 AS Utrecht, Niederlande

Handelsregister der Handelskammer Midden-Nederland, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher
Green Oracle <http://www.oracle.com/commitment> 	Oracle is committed 
to developing practices and products that help protect the environment





___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
ht

Re: ClassValue perf?

[Peter Levart]

Hi,


On 04/29/2016 10:28 AM, Michael Haupt wrote:

All,

see http://cr.openjdk.java.net/~mhaupt/8031043/ 
 for a snapshot of what 
is currently available.


We have three patches:
* Christian's, which simply reduces the HashMap size,
* Peter's, which refactors ClassValueMap into a WeakHashMap,
* mine, which attempts to introduce the single-value storage 
optimisation John had suggested (I worked on performance with Aleksey 
- thanks!).


All of these are collected in the patches subdirectory for 
convenience. (Peter, I adapted your patch to the new Unsafe location.)


I extended Peter's benchmark (thanks!) to cover single-value storage; 
the source code is in the benchmark subdirectory, together with raw 
results from running the benchmark with each of the three patches 
applied. A results-only overview is in benchmark-results.txt.


The three are roughly on par. I'm not sure the single-value storage 
optimisation improves much on footprint given the additional data that 
must be kept around to make transition to map storage safe.


Opinions?


I must admit that my old patch is very complex, so I doubt anyone will 
take time to review it. It is almost a clean-room re-implementation of 
ClassValue API. My main motivation was footprint optimization for all 
sizes - not just one value per class as I doubt this will be very common 
situation anyway. Current ClassValue maintains 2 parallel hash-tables 
per class. A WeakHashMap which is accessed with proper synchronization 
and an optimized "cache" of entries for quick access. This makes it 
consume almost 100 bytes per (Class, ClassValue) pair. I managed to 
almost half the overhead for typical situation (1024 classes x 16 
ClassValue(s)), but for the price of complexity.


Reviving this thread made me think about ClassValue again and I got 
another idea. This is an experiment to see if ConcurrentHashMap could be 
leveraged to implement ClassValue API with little added complexity:


http://cr.openjdk.java.net/~plevart/misc/ClassValue.Alternative2/webrev.01/

And here are the results of a benchmark comparing JDK 9 original with 
this alternative:


http://cr.openjdk.java.net/~plevart/misc/ClassValue.Alternative2/ClassValueBench.java

It is a little slower for random access of bigger sizes and #s of 
classes. Most probably a consequence of reduced cache hit ratio as CHM 
is a classical hash table with buckets implemented as linked list of 
entries whereas jdk 9 ClassValue cache is a linear-scan hash table which 
has better cache locality. This is particularly obvious in sequential 
access where CHM behaves on-par. It's a pity that CHM has a 
non-changeable load factor of 0.75 as changing this to 0.5 would most 
certainly improve benchmark results for a little more memory.


Where this version excels is in footprint. I managed to more than half 
the overhead. There's only a single ReferenceQueue needed and 
consequently expunging of stale data is more prompt and thorough. The 
code of ClassValue has been more than halved too.


What do you think?

Regards, Peter



Best,

Michael

--

Oracle 
Dr. Michael Haupt | Principal Member of Technical Staff
Phone: +49 331 200 7277 | Fax: +49 331 200 7561
OracleJava Platform Group | LangTools Team | Nashorn
Oracle Deutschland B.V. & Co. KG | Schiffbauergasse 14 | 14467 
Potsdam, Germany


ORACLE Deutschland B.V. & Co. KG | Hauptverwaltung: Riesstraße 25, 
D-80992 München

Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V. | Hertogswetering 
163/167, 3543 AS Utrecht, Niederlande

Handelsregister der Handelskammer Midden-Nederland, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher
Green Oracle  	Oracle is committed 
to developing practices and products that help protect the environment





___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: Questions about JDK-8136353 "ClassValue preventing class unloading"

[Peter Levart]

On 01/07/2016 02:18 AM, William ML Leslie wrote:

On 7 January 2016 at 07:34, Peter Levart <peter.lev...@gmail.com> wrote:

ClassValue API could use a different strategy and reference the cached
instance of type T from ClassValue instance instead, but then you would
have a problem with ClassValue instances reachable from other class loaders
than the cached instances of type T.

Regards, Peter

Is the problem with this strategy [together with a
java.util.WeakHashMap] that without this strong reference to the Class
object, the Class itself could be collected?



Hi William,

ClassValue API maps a pair (Class, ClassValue) -> cached instance. The 
cached instance is "retrievable" if and only if both parts of the pair 
(Class, ClassValue) can be produced. As soon as either Class key or 
ClassValue instance becomes unreachable, the cached instance could 
become unreachable too as it is clearly not "retrievable" any more. But 
the cached instance must be kept reachable while it is still possible to 
produce both parts of the pair (Class, ClassValue). So ClassValue API 
chooses to make a strong link from Class key to cached instance. 
Alternative strategy would be for API to make a strong link from 
ClassValue instance to cached instance. Either case has it's drawback.


The current ClassValue strategy has drawback when the cached instance 
directly or indirectly references the ClassValue instance it was 
constructed in (the case of the failing test in this thread). The cached 
instance lifetime is then bound to the lifetime of the Class key.


The alternative strategy has drawback when the cached instance directly 
of indirectly references the Class key it was constructed for. The 
cached instance lifetime is then bound to the lifetime of the ClassValue 
instance.


There's no way out short of choosing the right strategy for the task at 
hand. And I showed in the last post how to effectively transform current 
strategy to the alternative strategy by introducing an indirection to 
the cached value through a WeakReference and making a strong link from 
ClassValue instance to cache value at the same time.


Regards, Peter

P.S. With Ephemerons, I think the general solution would be possible.

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: Questions about JDK-8136353 "ClassValue preventing class unloading"

[Peter Levart]

On 01/07/2016 03:05 PM, Jochen Theodorou wrote:

Hi Peter,

Am 06.01.2016 um 21:34 schrieb Peter Levart:
[...]

Currently the cached instance of type T is reachable from Class
instance and you are using Integer.TYPE in your example. Which means
that the cached instance of T is never going to be released if
ClassValue instance is reachable from cached instance of T. In your
example, it is: (Dummy instance -> Dummy.class -> MyClassValue
instance). So you get this reachability chain: Integer.TYPE ->
ClassValueMap instance -> ClassValueMap.Entry[] -> ClassValueMap.Entry
-> Dummy instance -> Dummy.class -> MyClassValue instance.

>

Integer.TYPE is a Class instance representing int type which is
loaded by bootstrap class loader, which never goes away.


So in other words, this is supposed to work if I used for example 
Dummy instead of Integer.TYPE? Because this is not the case (just 
tested with 1.8.0_66). Which means we are not talking only about 
bootstrap classes, but about basically any class.


Hi Jochen,

I suspect you specified MyClassValue.Dummy.class instead of Integer.TYPE 
? Was Dummy.class loaded by same class loader as the MyClassValue for 
which the ClassValue.get() was executed. I have tried to execute the 
following variation of the test (no need to pack a separate jar file - 
you can execute if directly from within an IDE):


import java.io.File;
import java.io.UncheckedIOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.Paths;
import java.util.regex.Pattern;
import java.util.stream.Collectors;
import java.util.stream.Stream;

public class CVTest {

public static class MyClassValue extends ClassValue {
protected Object computeValue(Class type) {
Dummy ret = new Dummy();
Dummy.o = this;
return ret;
}
}

public static class Dummy {
static Object o;

@Override
protected void finalize() throws Throwable {
System.out.println(this + " finalized");
}
}

public static void main(String[] args) throws Exception {
String[] cp = System.getProperty("java.class.path")
.split(Pattern.quote(File.pathSeparator));
URL[] urls = Stream.of(cp).map(f -> {
try {
return Paths.get(f).toUri().toURL();
} catch (MalformedURLException e) {
throw new UncheckedIOException(e);
}
}).collect(Collectors.toList()).toArray(new URL[0]);

for (long i = 0; i < 10; i++) {
URLClassLoader classLoader = new URLClassLoader(
urls, ClassLoader.getSystemClassLoader().getParent());
ClassValue cv = (ClassValue)
classLoader.loadClass("CVTest$MyClassValue").newInstance();
Class key = classLoader.loadClass("CVTest$Dummy");
Object value = cv.get(key);
assert value.getClass().getClassLoader() == classLoader;
assert key.getClassLoader() == classLoader;
System.out.println(i);
classLoader.close();
classLoader = null;
cv = null;
key = null;
value = null;
System.gc();
Thread.sleep(200L);
}
}
}


...and it prints:

0
CVTest$Dummy@6b027549 finalized
1
CVTest$Dummy@65c59659 finalized
2
CVTest$Dummy@61fa1e39 finalized
3
CVTest$Dummy@24f89c62 finalized
4
CVTest$Dummy@5ca80d89 finalized
5
CVTest$Dummy@15963f1b finalized
6
CVTest$Dummy@242cf046 finalized
7
CVTest$Dummy@44d7632d finalized
8
CVTest$Dummy@588ffa7b finalized
9
CVTest$Dummy@7cc57896 finalized


...indicating that objects do get released in this case.


Regards, Peter




ClassValue API could use a different strategy and reference the cached
instance of type T from ClassValue instance instead, but then you
would have a problem with ClassValue instances reachable from other
class loaders than the cached instances of type T.


yes, that is what I tried as well... I used my own map directly in 
ClassValue and avoided the map ClassValue uses natively.


So the solution for this case is basically using a Soft- or 
Weak-Reference in Dummy, right?


bye Jochen
___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: Questions about JDK-8136353 "ClassValue preventing class unloading"

[Peter Levart]

Hi Craig,

I think that what you are asking for is impossible to achieve in current 
Java.


ClassValue API is basically a cache that maps a pair: (ClassValue, 
Class) -> cached instance of type T


You would like the cached instance of type T to not be reachable from 
either ClassValue instance or from Class instance, but still be 
non-reclaimable while either of ClassValue instance or Class 
instance is reachable in some other way that excludes being reachable 
from the cached instance of type T.


For that to be achievable, Java would have to provide a feature called 
"Ephemeron".


Currently the cached instance of type T is reachable from Class 
instance and you are using Integer.TYPE in your example. Which means 
that the cached instance of T is never going to be released if 
ClassValue instance is reachable from cached instance of T. In your 
example, it is: (Dummy instance -> Dummy.class -> MyClassValue 
instance). So you get this reachability chain: Integer.TYPE -> 
ClassValueMap instance -> ClassValueMap.Entry[] -> ClassValueMap.Entry 
-> Dummy instance -> Dummy.class -> MyClassValue instance.
Integer.TYPE is a Class instance representing int type which is 
loaded by bootstrap class loader, which never goes away.


ClassValue API could use a different strategy and reference the cached 
instance of type T from ClassValue instance instead, but then you 
would have a problem with ClassValue instances reachable from other 
class loaders than the cached instances of type T.


Regards, Peter

On 01/06/2016 06:14 PM, Craig Andrews wrote:
I apologize for contacting this list - I'm sure this isn't the "right 
way" to contact the OpenJDK project, but I'm not clear what the "right 
way" is.


I'm hoping to raise https://bugs.openjdk.java.net/browse/JDK-8136353 
"ClassValue preventing class unloading" as I believe it's a 
significant issue in ClassValue which is a blocker for its use in 
dynamic languages (which is primiarly why ClassValue was introduced). 
I think the P5 priority set on the bug now is way too low, perhaps you 
could consider raising the priority?


The source code in the issue description is incorrect; it doesn't 
compile. Could you please attach the working test cases to the issue, 
so future testers and developers can reproduce the problem? Here are 
links to the 2 source code files:

https://issues.apache.org/jira/secure/attachment/12765900/CVTest.java
https://issues.apache.org/jira/secure/attachment/12765901/MyClassValue.java 

(which of course should be directly attached to the openjdk issue 
tracker issue, and not hyperlinked to the Groovy tracker)


And to reproduce the issue, run:
$ javac MyClassValue.java && javac CVTest.java && mkdir -p t && jar 
cvf t/t.jar MyClassValue*.class && rm MyClassValue*.class && 
JAVA_OPTS=-Xmx4m java CVTest

and wait for the an error to occur, which is:
Exception in thread "main" java.lang.OutOfMemoryError: Compressed 
class space

at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:759)
at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:152)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:470)
at java.net.URLClassLoader.access$100(URLClassLoader.java:76)
at java.net.URLClassLoader$1.run(URLClassLoader.java:371)
at java.net.URLClassLoader$1.run(URLClassLoader.java:365)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:364)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at CVTest.main(CVTest.java:12)

The bug is reproducible on all JDK 8 and 9 builds (I tested up to JDK 
9 build 99).


Based on my understanding of the situation from my research in the 
Groovy bug that discovered the issue ( 
https://issues.apache.org/jira/browse/GROOVY-6704 ) and some work 
another person did with the YourKit profiler ( 
https://www.yourkit.com/forum/viewtopic.php?f=3=12658 ), I suspect 
that the fix for https://bugs.openjdk.java.net/browse/JDK-8032606 
"ClassValue.ClassValueMap.type is unused" is relevant; I think the 
problem lies in the java.lang.Class.classValueMap implementation.


Thank you,
~Craig Andrews
___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: Questions about JDK-8136353 "ClassValue preventing class unloading"

[Peter Levart]

On 01/06/2016 09:34 PM, Peter Levart wrote:

Hi Craig,

I think that what you are asking for is impossible to achieve in 
current Java.


ClassValue API is basically a cache that maps a pair: (ClassValue, 
Class) -> cached instance of type T


You would like the cached instance of type T to not be reachable from 
either ClassValue instance or from Class instance, but still be 
non-reclaimable while either of 

correction: while either of ... or ... is -> while both of ... and ... are

ClassValue instance or Class instance is reachable in some other 
way that excludes being reachable from the cached instance of type T.


For that to be achievable, Java would have to provide a feature called 
"Ephemeron".


Currently the cached instance of type T is reachable from Class 
instance and you are using Integer.TYPE in your example. Which means 
that the cached instance of T is never going to be released if 
ClassValue instance is reachable from cached instance of T. In your 
example, it is: (Dummy instance -> Dummy.class -> MyClassValue 
instance). So you get this reachability chain: Integer.TYPE -> 
ClassValueMap instance -> ClassValueMap.Entry[] -> ClassValueMap.Entry 
-> Dummy instance -> Dummy.class -> MyClassValue instance.
Integer.TYPE is a Class instance representing int type which is 
loaded by bootstrap class loader, which never goes away.


ClassValue API could use a different strategy and reference the cached 
instance of type T from ClassValue instance instead, but then you 
would have a problem with ClassValue instances reachable from other 
class loaders than the cached instances of type T.


Regards, Peter

On 01/06/2016 06:14 PM, Craig Andrews wrote:
I apologize for contacting this list - I'm sure this isn't the "right 
way" to contact the OpenJDK project, but I'm not clear what the 
"right way" is.


I'm hoping to raise https://bugs.openjdk.java.net/browse/JDK-8136353 
"ClassValue preventing class unloading" as I believe it's a 
significant issue in ClassValue which is a blocker for its use in 
dynamic languages (which is primiarly why ClassValue was introduced). 
I think the P5 priority set on the bug now is way too low, perhaps 
you could consider raising the priority?


The source code in the issue description is incorrect; it doesn't 
compile. Could you please attach the working test cases to the issue, 
so future testers and developers can reproduce the problem? Here are 
links to the 2 source code files:

https://issues.apache.org/jira/secure/attachment/12765900/CVTest.java
https://issues.apache.org/jira/secure/attachment/12765901/MyClassValue.java 

(which of course should be directly attached to the openjdk issue 
tracker issue, and not hyperlinked to the Groovy tracker)


And to reproduce the issue, run:
$ javac MyClassValue.java && javac CVTest.java && mkdir -p t && jar 
cvf t/t.jar MyClassValue*.class && rm MyClassValue*.class && 
JAVA_OPTS=-Xmx4m java CVTest

and wait for the an error to occur, which is:
Exception in thread "main" java.lang.OutOfMemoryError: Compressed 
class space

at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:759)
at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:152)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:470)
at java.net.URLClassLoader.access$100(URLClassLoader.java:76)
at java.net.URLClassLoader$1.run(URLClassLoader.java:371)
at java.net.URLClassLoader$1.run(URLClassLoader.java:365)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:364)
at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
at CVTest.main(CVTest.java:12)

The bug is reproducible on all JDK 8 and 9 builds (I tested up to JDK 
9 build 99).


Based on my understanding of the situation from my research in the 
Groovy bug that discovered the issue ( 
https://issues.apache.org/jira/browse/GROOVY-6704 ) and some work 
another person did with the YourKit profiler ( 
https://www.yourkit.com/forum/viewtopic.php?f=3=12658 ), I suspect 
that the fix for https://bugs.openjdk.java.net/browse/JDK-8032606 
"ClassValue.ClassValueMap.type is unused" is relevant; I think the 
problem lies in the java.lang.Class.classValueMap implementation.


Thank you,
~Craig Andrews
___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev




___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: RFR: JDK-8136893: Improve early java.lang.invoke infrastructure initialization

[Peter Levart]

Hi Paul,

Thanks for looking into this.

On 09/25/2015 04:07 PM, Paul Sandoz wrote:

Hi,

This looks like a partial dup of 
https://bugs.openjdk.java.net/browse/JDK-8076596


Ah, sorry, I wasn't aware this has already been registered in JIRA. I 
have linked the two issues together as DUPs.




The changes look ok, but I am concerned post initialization there may be code 
paths taken that require the system class loader to be used but instead the 
boot stream class loader is used instead. Is that a legitimate concern?


It is legitimate, but I have inspected usages and:

- In java.lang.invoke.BoundMethodHandle.Factory#makeCbmhCtor, null is 
passed explicitly and this method is used only from 
java.lang.invoke.BoundMethodHandle.Factory#makeCtors which is used from 
java.lang.invoke.BoundMethodHandle.SpeciesData#initForBootstrap and 
java.lang.invoke.BoundMethodHandle.SpeciesData#SpeciesData(java.lang.String, 
java.lang.Class). These 
usages only deal with erased MH types (Object, long, int, double, float).


- In java.lang.invoke.MemberName#getMethodType, the result of 
MemberName.getClassLoader() is passed to the method. This is the class 
loader of the member's declaring class. Any types referenced from the 
member declaration should be resolvable from this class loader. A member 
declared by a bootstrap class (MemberName.getClassLoader() returns null) 
can only reference types resolvable from bootstrap loader.


- In java.lang.invoke.MemberName#getFieldType, the result of 
MemberName.getClassLoader() is passed to the method. Similar applies as 
above.


- In java.lang.invoke.MethodHandleNatives#fixMethodType(Class 
callerClass, Object type), the callerClass.getClassLoader() is passed to 
the method. This is invoked from:

 java.lang.invoke.MethodHandleNatives#linkMethodImpl(
 Class callerClass, int refKind,
 Class defc, String name, Object 
type,

 Object[] appendixResult)
which is called from 
java.lang.invoke.MethodHandleNatives#linkMethod(same args)


I suppose this is an upcall from VM to link a call to the 
@PolymorphicSignature method and callerClass is the class in which the 
invocation bytecodes are executed. Am I right? The reasoning is as 
follows: if callerClass.getClassLoader() is sufficient for resolving 
types that appear at the call site in a non-bootstrap callerClass, then 
bootstrap classloader should be sufficient to resolve types that appear 
at the call site in a bootstrap callerClass. Does anybody have any other 
opinion?


- sun.invoke.util.BytecodeDescriptor#parseMethod(java.lang.String, 
java.lang.ClassLoader) is only used from the newly introduced 
java.lang.invoke.MethodType#fromDescriptor



Regards, Peter



Paul.

On 25 Sep 2015, at 15:37, Michael Haupt <michael.ha...@oracle.com> wrote:


Hi Peter,

thanks for this changeset. Note I'm not a Reviewer (with a capital R); please 
read this review in lower-case. ;-)

One question about MethodType: would you mind doing something about the naming 
of the newly introduced fromDescriptor() method? Its name does not suggest in 
any way that it chooses the class loader differently. The name is subtly 
different from that of fromDescriptorString(), and the signature is identical - 
it's probably really easy to confuse the two when working at the core libs 
level. Unfortunately, the only proposal for a name I can make, 
fromDescriptorStringBootCL(), is clunky. Maybe that's acceptable for a 
low-level method only visible at package level.


Am 25.09.2015 um 08:46 schrieb Peter Levart <peter.lev...@gmail.com>:
I have run the tests in java.lang.invoke and only have a problem with 1 test 
which seems to be related to the test or jtreg itself and happens also without 
my patch applied:

#Test Results (version 2)
#Tue Sep 22 09:48:38 CEST 2015
...
#section:script_messages
--messages:(0/0)--


test result: Error. Parse Exception: `@library' must appear before first `@run'

Yes. The test is also marked as ignored until another issue is fixed, so that 
can be ignored.

Other than the above remark/suggestion, this looks fine. I'll defer to an 
upper-case Reviewer, though.

Best,

Michael

--

<http://www.oracle.com/>
Dr. Michael Haupt | Principal Member of Technical Staff
Phone: +49 331 200 7277 | Fax: +49 331 200 7561
Oracle Java Platform Group | LangTools Team | Nashorn
Oracle Deutschland B.V. & Co. KG, Schiffbauergasse 14 | 14467 Potsdam, Germany
<http://www.oracle.com/commitment>Oracle is committed to developing 
practices and products that help protect the environment



___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: RFR: JDK-8136893: Improve early java.lang.invoke infrastructure initialization

[Peter Levart]

Hi Michael,

On 09/25/2015 03:37 PM, Michael Haupt wrote:

Hi Peter,

thanks for this changeset. Note I'm not a Reviewer (with a capital R); please 
read this review in lower-case. ;-)

One question about MethodType: would you mind doing something about the naming 
of the newly introduced fromDescriptor() method? Its name does not suggest in 
any way that it chooses the class loader differently. The name is subtly 
different from that of fromDescriptorString(), and the signature is identical - 
it's probably really easy to confuse the two when working at the core libs 
level. Unfortunately, the only proposal for a name I can make, 
fromDescriptorStringBootCL(), is clunky. Maybe that's acceptable for a 
low-level method only visible at package level.


Well, the correct self-describing name would then be 
fromDescriptorStringNullIsBootstrapCL()...


Perhaps a note in the javadoc that this is the preferred method for 
internal use would be sufficient? We can't make a reference from public 
method to the package-private one in javadoc though.


Another option would be to create new method as public API and 
@Deprecate the old one. There is a convention that null means bootstrap 
loader in other public APIs as well (for example: Class.forName(String 
name, boolean initialize, ClassLoader loader); ) although passing null 
from user code is rarely needed if at all. This is usually just for 
internal use.


Regards, Peter




Am 25.09.2015 um 08:46 schrieb Peter Levart <peter.lev...@gmail.com>:
I have run the tests in java.lang.invoke and only have a problem with 1 test 
which seems to be related to the test or jtreg itself and happens also without 
my patch applied:

#Test Results (version 2)
#Tue Sep 22 09:48:38 CEST 2015
...
#section:script_messages
--messages:(0/0)--


test result: Error. Parse Exception: `@library' must appear before first `@run'

Yes. The test is also marked as ignored until another issue is fixed, so that 
can be ignored.

Other than the above remark/suggestion, this looks fine. I'll defer to an 
upper-case Reviewer, though.

Best,

Michael



___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

RFR: JDK-8136893: Improve early java.lang.invoke infrastructure initialization

[Peter Levart]

Hi,

I propose a simple fix for two initialization issues described/linked in:

https://bugs.openjdk.java.net/browse/JDK-8136893

The patch is here:

http://cr.openjdk.java.net/~plevart/jdk9-dev/8136893_MethodType.fromDescriptor/webrev.01/

The 1st issue is the method: MetodType.fromMethodDescriptorString(String 
descriptor, ClassLoader loader)


This method is specified to treat null ClassLoader parameter as the 
system class loader. When lava.lang.invoke infrastructure is initialized 
before system class loader has been set, we get exceptions:


http://mail.openjdk.java.net/pipermail/mlvm-dev/2015-March/006386.html

java.lang.invoke initialization should not need to use system class 
loader as all types it must resolve during initialization are resolvable 
by bootstrap loader. Above method is public API and we can't change it's 
behavior, but can introduce another internal method that behaves a 
little differently - treating null ClassLoader as bootstrap class 
loader. All internal usages are rerouted to this internal method.


The 2nd issue is a little utility method: 
TypeConvertingMethodAdapter.boxingDescriptor(Wrapper w)


It uses String.format() to format a method type descriptor from two 
components. This unnecessarily brings in the Locale infrastructure which 
needs system class loader to locate providers:


http://mail.openjdk.java.net/pipermail/mlvm-dev/2015-March/006387.html

The fix is to replace String.format with simple string concatenation.

I have run the tests in java.lang.invoke and only have a problem with 1 
test which seems to be related to the test or jtreg itself and happens 
also without my patch applied:


#Test Results (version 2)
#Tue Sep 22 09:48:38 CEST 2015
#-testdescription-
$file=/home/peter/work/hg/jdk9-dev/jdk/test/java/lang/invoke/LFCaching/LFGarbageCollectedTest.java
$root=/home/peter/work/hg/jdk9-dev/jdk/test
author=kshefov
error=Parse Exception\: `@library' must appear before first `@run'
keywords=bug8046703 randomness othervm ignore
run=USER_SPECIFIED ignore 8078602\nUSER_SPECIFIED build 
TestMethods\nUSER_SPECIFIED build LambdaFormTestCase\nUSER_SPECIFIED 
build LFGarbageCollectedTest\nUSER_SPECIFIED main/othervm -Xmx64m 
-XX\:SoftRefLRUPolicyMSPerMB\=0 -XX\:+HeapDumpOnOutOfMemoryError 
-DHEAP_DUMP\=false LFGarbageCollectedTest\n

source=LFGarbageCollectedTest.java
title=Test verifies that lambda forms are garbage collected

#-environment-

#-testresult-
description=file\:/home/peter/work/hg/jdk9-dev/jdk/test/java/lang/invoke/LFCaching/LFGarbageCollectedTest.java
elapsed=2 0\:00\:00.002
end=Tue Sep 22 09\:48\:38 CEST 2015
environment=regtest
execStatus=Error. Parse Exception\: `@library' must appear before first 
`@run'

harnessLoaderMode=Classpath Loader
harnessVariety=Full Bundle
hostname=peterl.marand.si
javatestOS=Linux 4.1.5-100.fc21.x86_64 (amd64)
javatestVersion=4.6
jtregVersion=jtreg 4.2.0 dev tip
script=com.sun.javatest.regtest.RegressionScript
sections=script_messages
start=Tue Sep 22 09\:48\:38 CEST 2015
test=java/lang/invoke/LFCaching/LFGarbageCollectedTest.java
totalTime=2
user.name=peter
work=/home/peter/work/hg/jdk9-dev/jdk/JTwork/java/lang/invoke/LFCaching

#section:script_messages
--messages:(0/0)--


test result: Error. Parse Exception: `@library' must appear before first 
`@run'




Regards, Peter

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: ClassLoader leak in MethodHandle.asType()

[Peter Levart]

On 04/25/2015 02:12 AM, John Rose wrote:
On Apr 24, 2015, at 3:24 PM, Peter Levart peter.lev...@gmail.com 
mailto:peter.lev...@gmail.com wrote:


Anyway. The inexact invoke() always transforms a specific MH to a 
generic one (Object, Object, ...)Object, right?


Yes.

So using inexact invoke() on any MH can't trigger the leak. It's just 
that if someone attempts to transform a generic MH to some more 
concrete one to be able to connect it with further transformations 
does the danger of the leak appear.


Good point, thanks.

This suggests that the right answer is to keep the 1-element cache as-is,
but don't fill it with any mh1=mh0.asType(...) for which mh1.type 
introduces

a class loader not already present in mh0.type.

I.e., don't use a weak reference for the 1-element cache, but don't 
fill it

with anything that might require a weak reference.  Use a backup cache
(if needed) to handle exotic type changes.

So maybe, just for the purpose of inexact invoke(), the caching could 
be performed just in case when asType transformed method type is of 
the form (Object, Object, ...)Object. This would also prevent 
accidental thrashing of the cache when inexact invoke() is intermixed 
with other asType() invocations.


The 1-element cache thrashes for other reasons already, and
I think we need a multi-way (multi-MH) backup cache of some sort.
The backup cache would have weak references.

Having a 1-element front-side cache would cover inefficiencies
in the backup cache, which in turn would cover inefficiencies
for generating new conversion LFs and/or MHs.

One interesting point:  If we change the 1-element cache to use
a weak reference, the cost of reloading the cache will be higher,
since each reload must create a new WeakReference instance.
This is a possible reason to hold off on the suggested fix.

— John


I thought so, yes. But checking for the cache-ability might not be free 
either. OTOH asTypeUncached() logic is not trivial as it is. It already 
includes checks like isConvertibleTo, etc, so another check might not be 
costly relatively. And not introducing another de-reference 
(WeakReference) makes the fast-path (the cache hit) unaffected.


Something like the following then?

http://cr.openjdk.java.net/~plevart/jdk9-dev/MethodHandle.asTypeCacheLeak/webrev.02/

Regards, Peter




___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: ClassLoader leak in MethodHandle.asType()

[Peter Levart]

On 04/24/2015 11:06 PM, John Rose wrote:

Good point.  Are you seeing a leak in practice?
The cache is important, especially to inexact MH.invoke.
— John


Well, yes. I am (re)implementing annotations (proxies) using Remi's 
Proxy2 and made it all the way except for the following failing jtreg test:


jdk/test/java/lang/annotation/loaderLeak/Main.java

...which made me find this issue. What I'm doing is the following:

For implementing some annotation @Ann proxy's hashCode() method for 
example, I prepare DMHs with the following signature (T)int for Ts being 
all primitives, primitive arrays, Object[] and Object. I cache them in a 
system class static field referenced HashMap keyed by ClassT. These 
are the functions that take an annotation member value and map it to 
it's hashCode. When the value is of some Object (or Object[] in case it 
is an array) subtype - for example when the member value is an 
annotation of type V, I take the (Object)int DMH and transform it with 
.asType(methodType(int.class, V.class)) to get (V)int, which is 
necessary to further wrap it with filterArguments where the filter is a 
member field getter of type (Ann)V so that I get (Ann)int MHs as a kind 
of hashCodeExtractors for all annotation members (I'll present the 
details later)...


Now if V is an annotation type that is loaded by a child class loader, 
it gets retained by system cached DMH.


But now that I'm writing this, I see a workarround. Instead of 
transforming hashCode function (Object)int - (V)int and applying field 
getter (Ann)V as an argument filter, I could transform the field getter 
argument filter (Ann)V - (Ann)Object and apply it directly to the 
argument of hashCode function (Object)int.


Anyway. The inexact invoke() always transforms a specific MH to a 
generic one (Object, Object, ...)Object, right? So using inexact 
invoke() on any MH can't trigger the leak. It's just that if someone 
attempts to transform a generic MH to some more concrete one to be able 
to connect it with further transformations does the danger of the leak 
appear. So maybe, just for the purpose of inexact invoke(), the caching 
could be performed just in case when asType transformed method type is 
of the form (Object, Object, ...)Object. This would also prevent 
accidental thrashing of the cache when inexact invoke() is intermixed 
with other asType() invocations.


Regards, Peter



On Apr 24, 2015, at 9:14 AM, Peter Levart peter.lev...@gmail.com 
mailto:peter.lev...@gmail.com wrote:


I think there is a Class(Loader) leak in MethodHandle.asType() 
implementation. If for example some MH (say mhX) is long lived 
(because it is a system cached MH) and a client generates variants 
from it by invoking mhX.asType(newType) and the newType contains a 
type (either return type or parameter type) that is loaded by some 
child ClassLoader, then such derived MH is retained strongly from mhX 
via the MethodHandle.asTypeCache field. Until this field is 
overwriten by some other MH, child ClassLoader can not be GCed.


In case this one-element cache is needed to speed things up, it 
should be a WeakReference, like the following:


http://cr.openjdk.java.net/~plevart/jdk9-dev/MethodHandle.asTypeCacheLeak/webrev.01/ 
http://cr.openjdk.java.net/%7Eplevart/jdk9-dev/MethodHandle.asTypeCacheLeak/webrev.01/






___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

ClassLoader leak in MethodHandle.asType()

[Peter Levart]

Hi,

I think there is a Class(Loader) leak in MethodHandle.asType() 
implementation. If for example some MH (say mhX) is long lived (because 
it is a system cached MH) and a client generates variants from it by 
invoking mhX.asType(newType) and the newType contains a type (either 
return type or parameter type) that is loaded by some child ClassLoader, 
then such derived MH is retained strongly from mhX via the 
MethodHandle.asTypeCache field. Until this field is overwriten by some 
other MH, child ClassLoader can not be GCed.


In case this one-element cache is needed to speed things up, it should 
be a WeakReference, like the following:


http://cr.openjdk.java.net/~plevart/jdk9-dev/MethodHandle.asTypeCacheLeak/webrev.01/


Regards, Peter

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: Implementing VarHandle

[Peter Levart]

On 04/19/2015 02:06 PM, Brian Goetz wrote:

Thanks Remi.

I’d like to separate this discussion into two components: implementation and 
API.  Let’s talk about API first.  We did give a fair amount of thought to the 
sigpoly-vs-typesafe API question.

VarHandles allow you to abstract data access over several dimensions:
  - location kind: static field, instance field, array element, native pointer, 
etc
  - variable type: int, long, Object, etc
  - access kind: relaxed, ordered, volatile, etc

Both your approach and ours follow the same route for access kind: separate 
methods for each kind of access.

Your approach unrolls the variable type dimension as well; this is a tradeoff 
of client-side type-safety for API surface area.  There’s a valid discussion to 
be had here about the pros and cons of each.

Where the biggest difference is in location kind.  Your approach is built 
around “instance field” as being the preferred location kind, and levering the 
rest into that shape (i.e., use null as receiver for static fields.)  This is 
obviously great for instance fields, acceptable for static fields, bad for 
array elements, and probably not so good for native access.  To really get type 
safety, we’d probably have to add new interfaces for 
{Static,Instance,Array,Native}VarHandle, further increasing the surface area of 
the API.  Also, this is very different from the approach taken by MH, where 
there is a single polymorphic invoke method, rather than n*m*k methods for 
different combinations of (kind, type, access).

The thing that pushed us over the edge is that value types are coming.  With value 
types, one can create type-safe, zero-cost, specialized wrappers for 
{Static,Instance,Array,Native}VarHandleT that wrap an underlying VH; because 
these wrappers can be values, they can provide type safety with no indirection or 
footprint costs.  So it seemed better to provide a simple, clean, low-level API now 
that doesn’t make any assumptions, let the early adopters (mostly us) deal with the 
fact that type safety comes at runtime (just as with MHs), and later provide a clean 
set of value wrappers on top of it.


This seems like a good plan for post-JDK9 times. But I still miss one 
thing in this picture - the syntax. If purely API approach is taken, 
then we will still be using Strings to identify fields and do the 
caching of VarHandles ourselves. Are there any plans for specifying 
syntax for constant [Method|Var] handles in Java or is this being 
reserved for post-JDK9 times where the syntax will be used to produce 
type-safe wrappers (similar to approach taken with MethodHandles vs. 
Lambdas)?


Regards, Peter


On Apr 12, 2015, at 4:54 PM, Remi Forax fo...@univ-mlv.fr wrote:


Hi guys,
I was about to write a blog post explaining why i don't like the way VarHandle 
are currently implemented when it occurs to me that providing another 
implementation may be a more efficient to discuss about implementation.

So my implementation is here,
  https://github.com/forax/varhandle2

the API is typesafe, the implementation is runtime safe and i get mostly the 
same assembly code using unsafe or this VarHandle API.

The idea is that there is no need to add more polymorphic signature methods, a 
good old Java method is enough if it using a method handle under the hood.
All the logic is described using method handles and the only logic written in 
Java is
1) when to create such method handle (the first time a method is called),
it works like invokedynamic but the method handles are stable instead of 
being constant.
2) how to link all things together to do nullcheck, atomic operation and post 
operation (like fence or sum).

cheers,
Rémi




___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: [9] RFR (M): 8057967: CallSite dependency tracking scales devastatingly poorly

[Peter Levart]

Hi Vladimir,

Would it be possible for CallSite.context to hold the Cleaner instance 
itself (without indirection through DependencyContext)?


DEFAULT_CONTEXT would then be a Cleaner instance that references some 
default Class object (for example DefaultContext.class that serves no 
other purpose).


Regards, Peter

On 04/01/2015 10:56 PM, Vladimir Ivanov wrote:

http://cr.openjdk.java.net/~vlivanov/8057967/webrev.00/hotspot/
http://cr.openjdk.java.net/~vlivanov/8057967/webrev.00/jdk/
https://bugs.openjdk.java.net/browse/JDK-8057967

HotSpot JITs inline very aggressively through CallSites. The 
optimistically treat CallSite target as constant, but record a nmethod 
dependency to invalidate the compiled code once CallSite target changes.


Right now, such dependencies have call site class as a context. This 
context is too coarse and it leads to context pollution: if some 
CallSite target changes, VM needs to enumerate all nmethods which 
depends on call sites of such type.


As performance analysis in the bug report shows, it can sum to 
significant amount of work.


While working on the fix, I investigated 3 approaches:
  (1) unique context per call site
  (2) use CallSite target class
  (3) use a class the CallSite instance is linked to

Considering call sites are ubiquitous (e.g. 10,000s on some octane 
benchmarks), loading a dedicated class for every call site is an 
overkill (even VM anonymous).


CallSite target class 
(MethodHandle.form-LambdaForm.vmentry-MemberName.clazz-Class?) is 
also not satisfactory, since it is a compiled LambdaForm VM anonymous 
class, which is heavily shared. It gets context pollution down, but 
still the overhead is quite high.


So, I decided to focus on (3) and ended up with a mixture of (2)  (3).

Comparing to other options, the complications of (3) are:
  - CallSite can stay unlinked (e.g. CallSite.dynamicInvoker()), so 
there should be some default context VM can use


  - CallSite instances can be shared and it shouldn't keep the context 
class from unloading;


It motivated a scheme where CallSite context is initialized lazily and 
can change during lifetime. When CallSite is linked with an indy 
instruction, it's context is initialized. Usually, JIT sees CallSite 
instances with initialized context (since it reaches them through 
indy), but if it's not the case and there's no context yet, JIT sets 
it to default context, which means use target call site.


I introduced CallSite$DependencyContext, which represents a nmethod 
dependency context and points (indirectly) to a Class? used as a 
context.


Context class is referenced through a phantom reference 
(sun.misc.Cleaner to simplify cleanup). Though it's impossible to 
extract referent using Reference.get(), VM can access it directly by 
reading corresponding field. Unlike other types of references, phantom 
references aren't cleared automatically. It allows VM to access 
context class until cleanup is performed. And cleanup resets the 
context to NULL, in addition to invalidating all relevant dependencies.


There are 3 context states a CallSite instance can be in:
  (1) NULL: no depedencies
  (2) DependencyContext.DEFAULT_CONTEXT: dependencies are stored in 
call site target class
  (3) DependencyContext for some class: dependencies are stored on the 
class DependencyContext instance points to


Every CallSite starts w/o a context (1) and then lazily gets one ((2) 
or (3) depending on the situation).


State transitions:
  (1-3): When a CallSite w/o a context (1) is linked with some indy 
call site, it's owner is recorded as a context (3).


  (1-2): When JIT needs to record a dependency on a target of a 
CallSite w/o a context(1), it sets the context to DEFAULT_CONTEXT and 
uses target class to store the dependency.


  (3-1): When context class becomes unreachable, a cleanup hook 
invalidates all dependencies on that CallSite and resets the context 
to NULL (1).


Only (3-1) requires dependency invalidation, because there are no 
depedencies in (1) and (2-1) isn't performed.


(1-3) is done in Java code (CallSite.initContext) and (1-2) is 
performed in VM (ciCallSite::get_context()). The updates are performed 
by CAS, so there's no need in additional synchronization. Other 
operations on VM side are volatile (to play well with Java code) and 
performed with Compile_lock held (to avoid races between VM operations).


Some statistics:
  Box2D, latest jdk9-dev
- CallSite instances: ~22000

- invalidated nmethods due to CallSite target changes: ~60

- checked call_site_target_value dependencies:
  - before the fix: ~1,600,000
  - after the fix:~600

Testing:
  - dedicated test which excercises different state transitions
  - jdk/java/lang/invoke, hotspot/test/compiler/jsr292, nashorn

Thanks!

Best regards,
Vladimir Ivanov


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: What can we improve in JSR292 for Java 9?

[Peter Levart]

On 02/26/2015 01:02 AM, Charles Oliver Nutter wrote:

After talking with folks at the Jfokus VM Summit, it seems like
there's a number of nice-to-have and a few need-to-have features we'd
like to see get into java.lang.invoke. Vladimir suggested I start a
thread on these features.

A few from me:

* A loop handle :-)

Given a body and a test, run the body until the test is false. I'm
guessing there's a good reason we don't have this already.

* try/finally as a core atom of MethodHandles API.

Libraries like invokebinder provide a shortcut API To generating the
large tree of handles needed for try/finally, but the JVM may not be
able to optimize that tree as well as a purpose-built adapter.

* Argument grouping operations in the middle of the argument list.

JRuby has many signatures that vararg somewhere other than the end of
the argument list, and the juggling required to do that logic in
handles is complex: shift to-be-boxed args to end, box them, shift box
back.

Another point about these more complicated forms: they're ESPECIALLY
slow early in execution, before LFs have been compiled to bytecode.

* Implementation-specific inspection API.

I know there are different ways to express a MH tree on different JVMs
(e.g. J9) but it would still be a big help for me if there were a good
way to get some debug-time structural information about a handle I'm
using. Hidden API would be ok if it's not too hidden :-)

That's off the top of my head. Others?

- Charlie


Hi,

What would be nice to have is support for constant direct method handles 
and invokedynamic in Java language. We already have a syntax (thanks to 
lambdas) and target typing. What I'm asking for is for the following...


constant direct method handles:

class C {
int someMethod(String s);
}

...

MethodHandle mh = C::someMethod(String);

No pre-bound MHs, no overloaded method resolution. Just plain 
compilation to MH constant load.



invokedynamic:

There already was support for this in JSR292 prototype. Why was it removed?


If for nothing else, this would simplify writing tests and experiments. 
Currently all we have is bytecode spinning with tools like ASM.



Regards, Peter


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: What can we improve in JSR292 for Java 9?

[Peter Levart]

On 03/08/2015 01:38 PM, Jochen Theodorou wrote:

Am 08.03.2015 12:16, schrieb Remi Forax:
[...]

You need to restrict the set of method handles that are allowed to be
installed inside a CallSite corresponding to such kind of invokedynamic.
Something like: you can do any transformations you want on the arguments
but not on the receiver (no drop, no permute, no filter) and then you
need to call unconditionally a method handle created using findSpecial
on the super class.


unconditionally, like no guard?


@Jochen:

There can be GWT, but it can only select one of target/fallback that are 
both from the restricted set of MHs.


@Remi:

That was my thinking too. I even started a proof-of-concept. The idea is 
as follows:


1) let verifier treat an invokedynamic super-init using a special 
system provided bootstrap method in the same way as it treats 
invokespecial super.init, meaning that such chaining to super 
constructor passes the verification

2) this system provided bootstrap method takes the standard parameters:
- caller Lookup
- invoked method name
- invoked method type
- and an additional parameter, a MethodHandle of a static method 
that is supplied by language runtime and has the purpose to build a MH 
chain that dispatches the invocation to one of candidate super 
constructors selected by system provided bootstrap method


I started a proof-of-concept that tries to implement a similar setup, 
but instead of invoking super constructor, it invokes one of accessible 
superclass instance methods that have the same name as dynamically 
invoked name - this should translate to constructors nicely, I think.


Here's a preview of what I'm trying to do:

http://cr.openjdk.java.net/~plevart/misc/invoke.Story/Story.java


The Story.superInvokeBootstrap is an example of such system provided 
bootstrap method that is trusted to do the right thing and treated by 
verifier in a special way. It prepares a set of target method handles 
and wraps them in wrapper objects that provide an API similar to 
MethodHandle/MethodHandles (Story.MH/Story) but restricted in a way that 
allows only transformations that maintain an invariant where the 1st 
argument (the target of instance method) is left untouched and invoking 
any of derived MHs results in either throwing an exception or invokes 
one of the provided candidate MHs.


The language runtime must therefore provide a static MH-chain building 
method similar to the following:


public static Story.MH buildDispatch(
MethodHandles.Lookup callerLookup,
String methodName, // the invoked method name
MethodType methodType, // the invoked type
Story story,
ListStory.MH candidates) {

...
 }

which uses Story.MH/Story API in a way that MethodHandle/MethodHandles 
API is used to construct and return a Story.MH that does the dispatch 
based on all arguments.


So Jochen, can you point me to an algorithm used for MultiMethods 
dispatch in Groovy and I'll try to prepare a proof-of-concept that uses it?



Regards, Peter




bye Jochen




___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: What can we improve in JSR292 for Java 9?

[Peter Levart]

On 03/07/2015 02:53 PM, Remi Forax wrote:


On 03/07/2015 06:31 AM, John Rose wrote:
[...]



(I wish we had a similar candidate for invokespecial/super. That is 
badly twisted around the verifier.)


One way to solve the problem is to consider that invokedynamic init 
is a special 'bytecode'
for the verifier and that the verifier will allow to call it on an 
non-initialized reference and

consider the reference as been initialized after that.

Rémi


As I understand the problem is security-related. By mandating that each 
constructor chains to a super constructor (or one of sibling 
constructors that finally calls super), a class can rely on the fact 
that it's state is initialized using one of accessible constructors. The 
verifier makes sure there can be no subclass that doesn't call one of 
super constructors as part of object construction. Some security-related 
idioms are based on that guarantee. For example: services looked-up 
using ServiceLoader typically use an abstract class as the service 
interface that implementations must extend. The abstract class 
constructor checks for permissions. Each service implementation subclass 
must call the super constructor and the permission check in it has the 
subclass code on the call-stack when making a permission check.


So how can verifier be sure that invokedynamic super-init eventually 
invokes one of the super constructors?


Peter



___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: What can we improve in JSR292 for Java 9?

[Peter Levart]

On 03/05/2015 04:09 AM, Jochen Theodorou wrote:

Am 04.03.2015 23:50, schrieb Charles Oliver Nutter:
On Thu, Feb 26, 2015 at 4:27 AM, Jochen Theodorou blackd...@gmx.org 
wrote:

my biggest request: allow the call of a super constructor (like
super(foo,bar)) using MethodHandles an have it understood by the JVM 
like a

normal super constructor call... same for this(...)


Just so I understand...the problem is that unless you can get a Lookup
that can do the super call from Java (i.e. from within a subclass),
you can't get a handle that can do the super call, right? And you
can't do that because the method bodies might not be emitted into a
natural subclass of the super class?


Not fully right. Sorry for the possibly false flag with the 
permissions from the runtime. I am in a natural subclass. Take this 
pseudoexample:


public class Foo {
  public Foo(String s, Integer i){}
  public Foo(Integer s, Object o){}
}

public class Bar extends Foo {
  public Bar(def a, def b) {
 super(a,b) // call with runtime types here
  }
}

I cannot express super(a,b) using method handles, even if I did know 
the types at compiletime (in which case I would not need invokedynamic 
anyway). Sure, there is 
http://docs.oracle.com/javase/7/docs/api/java/lang/invoke/MethodHandles.Lookup.html#findSpecial%28java.lang.Class,%20java.lang.String,%20java.lang.invoke.MethodType,%20java.lang.Class%29 
but this is for calling methods in a invokespecial like manner, init 
is excluded here as stated in the comment. The comment refers to 
findConsructor for calls to constructors, and indeed we have 
invokeSpecial calls there as well, but those are for what in Java is 
new, and are different from calls to a super constructor. See also 
http://mail.openjdk.java.net/pipermail/mlvm-dev/2012-June/004650.html 
from almost 3 years ago here on the list. Remi gives here a good 
reason why it probably cannot be done easily... in that or an even 
older thread someone (I think John) was talking about security.


bye Jochen



Hi Jochen,

Here's an idea. Let Groovy compile Foo to the following Java equivalent:

public class Foo {
public Foo(Onject p1, Object p2) {
super();
invokedynamic _init(p1, p2);
}

private void _init(String s, Integer i) {
// the body of constructor for (String, Integer)
}

private void _init(Integer s, Object o) {
// the body of constructor for (Integer, Object)
}
}


... a set of overloaded Groovy constructors with same arity could be 
emmited as:


- a single constructor with Object typed arguments chaining to a super 
constructor with Object typed arguments using plain invokespecial
- one private void _init method with declared types of arguments per 
Groovy constructor containing code of related Groovy constructor
- an invokedynamic call in the single emitted constructor used for 
dispatching to appropriate _init method based on runtime types of arguments


Only one thing would not work as expected in this setting: final fields 
could not be set in _init methods as verifier doesn't allow that.


So what we might need (also for other purposes like de-serialization) is 
a special kind of private void instance initialization methods that are 
treated specially by verifier and javac. The rules for such methods 
could be as follows:


- they are treated like constructors regarding assignment to final 
instance fields
- they can not be called from normal code except from constructors of 
the same class that calls: super constructor followed by a call to one 
of those special initialization methods. For example:


public class Bar extends Foo {

private final int val;

public Bar(String s, int val) {
super(s);
// call to special @init method can only appear immediately 
after call to super constructor (verifier checked)
initVal(val); // ...together they have the effect of calling 
any this(...) constructor


// ...so this is not allowed by javac
this.val = 42;
}

@init private void initVal(int val) {
this.val = val; // allowed and required (like in constructor)
}

public void normalMethod() {
initVal(42); // not allowed by javac or verifier
}
...


Those special @init methods could be invoked using reflection with 
overridden access checks (setAccessible(true)) and looked up as method 
handles using privileged Lookup only.



Regards, Peter

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: [9] RFR (M): 8069591: Customize LambdaForms which are invoked using MH.invoke/invokeExact

[Peter Levart]

On 01/23/2015 12:30 AM, John Rose wrote:

On Jan 22, 2015, at 9:56 AM, Vladimir Ivanov vladimir.x.iva...@oracle.com 
wrote:

Remi, John, thanks for review!

Updated webrev:
http://cr.openjdk.java.net/~vlivanov/8069591/webrev.01/

This time I did additional testing (COMPILE_THRESHOLD  0) and spotted a 
problem with MethodHandle.copyWith(): a MethodHandle can inherit customized 
LambdaForm this way. I could have added LambdaForm::uncustomize() call in evey 
Species_*::copyWith() method, but I decided to add it into MethodHandle 
constructor. Let me know if you think it's too intrusive.

It's OK to put it there.

Now I'm worried that the new customization logic will defeat code sharing for 
invoked MHs, since uncustomize creates a new LF that is a duplicate of the 
original LF.  That breaks the genetic link for children of the invoked MH, 
doesn't it?  (I like the compileToBytecode call, if it is done on the 
original.)  In fact, that is also a potential problem for the first version of 
your patch, also.

Suggestion:  Have every customized LF contain a direct link to its uncustomized 
original.  Have uncustomize just return that same original, every time.  Then, 
when using LF editor operations to derive new LFs, always have them extract the 
original before making a derivation.


The customized LF then don't need 'transformCache' field. It could be 
re-used to point to original uncustomized LF. That would also be a 
signal for LF editor (the 4th type of payload attached to transformCache 
field) to follow the link to get to the uncustomized LF...


Peter



(Alternatively, have the LF editor caches be shared between original LFs and 
all their customized versions.  But that doesn't save all the genetic links.)


Also, I made DirectMethodHandles a special-case, since I don't see any benefit 
in customizing them.

The overriding method in DHM should be marked @Override, so that we know all 
the bits fit together.

— John
___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: [9] RFR (M): 8067344: Adjust java/lang/invoke/LFCaching/LFGarbageCollectedTest.java for recent changes in java.lang.invoke

[Peter Levart]

Hi Vladimir,

I just wanted to ask about the reason you used PhantomReference to 
detect that a referent is not softly reachable any more. You could use 
another SoftReference or a WeakReference for the same effect, since it 
is guaranteed that all SoftReferences for same referent are cleared 
atomically and WeakReferences are cleared when referent is not softly 
reachable any more. Using Weak or PhantomReference in test can 
theoretically give false failures, since the object can still be weakly 
of phantom reachable, but not softly reachable any more.


I think that it is enough for test to check that the referent is not 
softly reachable any more, since cache is using SoftReferences. 
Therefore it would be most appropriate to use a SoftReference in test too.


What do you think?

Regards, Peter

On 12/23/2014 01:40 PM, Vladimir Ivanov wrote:

Spotted some more problems:
  - need to skip identity operations (identity_* LambdaForms) in the 
test, since corresponding LambdaForms reside in a permanent cache;


  - need to keep original test data for diagnostic purposes, since 
getTestCaseData() produces new instance.


Updated version:
http://cr.openjdk.java.net/~vlivanov/8067344/webrev.01/

Best regards,
Vladimir Ivanov

On 12/22/14 11:53 PM, Vladimir Ivanov wrote:

http://cr.openjdk.java.net/~vlivanov/8067344/webrev.00/
https://bugs.openjdk.java.net/browse/JDK-8067344

LFGarbageCollectedTest should be adjusted after JDK-8057020.

There are a couple of problems with the test.

(1) Existing logic to test that LambdaForm instance is collected isn't
stable enough. Consequent System.GCs can hinder reference enqueueing.
To speed up the test, I added -XX:SoftRefLRUPolicyMSPerMB=0 and limited
the heap by -Xmx64m.

(2) MethodType-based invoker caches are deliberately left strongly
reachable. So, they should be skipped in the test.

(3) Added additional diagnostic output to simplify failure analysis
(test case details, method handle type and LambdaForm, heap dump
(optional, -DHEAP_DUMP=true)).

Testing: failing test.

Thanks!

Best regards,
Vladimir Ivanov


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: [9, 8u40] RFR (M): 8057020: LambdaForm caches should support eviction

[Peter Levart]

Just one more thought...

In lines:

 358 } else if (stale  0  k.get() == null) {
 359 stale = i; // remember 1st stale entry index
 360 }


...an index to 1st stale entry is remembered while scanning the array so 
that at the end, instead of always allocating new slot, an existing slot 
can be re-used. This has an inadvertent effect on the lifetime of 
SoftReference(s) that are touched in the process (LRU policy of 
cleaning). Here we see that an additional method like 
SoftReference.isPresent() that didn't touch the access time, would be 
helpful.


Regards, Peter

On 12/06/2014 01:30 PM, Peter Levart wrote:

(Sorry for a re-send, forgot to include core-libs-dev)...

Hi Vladimir,

First, just a nit. I think that in LambdaFormEditor:

 289 private LambdaForm putInCache(Transform key, LambdaForm form) {
 290 key = key.withResult(form);
 291 for (int pass = 0; ; pass++) {
 292 Object c = lambdaForm.transformCache;
 293 if (c instanceof ConcurrentHashMap) {
 294 @SuppressWarnings(unchecked)
 295 ConcurrentHashMapTransform,Transform m = 
(ConcurrentHashMapTransform,Transform) c;

 296 Transform k = m.putIfAbsent(key, key);
 297 if (k == null) return form;
 298 LambdaForm result = k.get();
 299 if (result != null) {
 300 return result;
 301 } else {
 302 if (m.replace(key, k, key)) {
 303 return form;
 304 } else {
 305 continue;
 306 }
 307 }
 308 }
 309 assert(pass == 0);
 310 synchronized (lambdaForm) {
 311 c = lambdaForm.transformCache;
 312 if (c instanceof ConcurrentHashMap)
 313 continue;

...

 372 lambdaForm.transformCache = c = m;
 ^^^ put 
assignment to 'c' back in
 373 // The second iteration will update for this 
query, concurrently.

 374 continue;


...you could move the assignment to 'c' in line 292 out of for loop 
and put it back in line 372, since once 'c' is instance of CHM, 
lambdaForm.transformCache never changes again and if 'c' is not CHM 
yet, it is re-assigned in lines 311 and 372 before next loop.


Am I right?

Now what scares me (might be that I don't have an intimacy with 
LambdaForm class like you do). There is a situation where you publish 
LambdaForm instances via data race.


One form of LambdaForm.transformCache is an array of Transform objects 
(the other two forms are not problematic). Transform class has all 
fields final except the 'referent' field of SoftReference, which holds 
a LambdaForm instance. In the following line:


377 ta[idx] = key;


...you publish Transform object to an element of array with relaxed 
write, and in the following lines:


 271 } else {
 272 Transform[] ta = (Transform[])c;
 273 for (int i = 0; i  ta.length; i++) {
 274 Transform t = ta[i];
 275 if (t == null)  break;
 276 if (t.equals(key)) { k = t; break; }
 277 }
 278 }
 279 assert(k == null || key.equals(k));
 280 return (k != null) ? k.get() : null;


...you obtain the element of the array with no synchronization and a 
relaxed read and might return a non-null referent (the LambdaForm) 
which is then returned as an interned instance.


So can LambdaForm instances be published via data races without fear 
that they would appear half-initialized?


That's what I didn't know when I used a lazySet coupled with volatile 
get to access array elements in my version:


http://cr.openjdk.java.net/~plevart/misc/LambdaFormEditor.WeakCache/webrev.01/ 




Regards, Peter

On 12/03/2014 12:45 PM, Vladimir Ivanov wrote:

Aleksey, thanks for the review.

I haven't tried -XX:SoftRefLRUPolicyMSPerMB=0, but I did extensive 
testing on Octane/Nashorn with multiple low -Xmx levels + frequent 
Full GCs (8060147 [1] was the result of those experiments) and stress 
tested cache eviction with jdk/java/lang/invoke/LFCache tests in long 
running mode.


Best regards,
Vladimir Ivanov

[1] https://bugs.openjdk.java.net/browse/JDK-8060147

On 12/3/14, 3:11 PM, Aleksey Shipilev wrote:

On 12/01/2014 07:58 PM, Vladimir Ivanov wrote:

http://cr.openjdk.java.net/~vlivanov/8057020/webrev.00/
https://bugs.openjdk.java.net/browse/JDK-8057020


Looks okay, although the cache management logic gives me a headache
after the vacation. I thought I spotted a few bugs, but those were only
false positives.


The fix is to use SoftReferences to keep LambdaForms alive as long as
possible, but avoid throwing OOME until the caches are evicted. I

Re: [9, 8u40] RFR (M): 8057020: LambdaForm caches should support eviction

[Peter Levart]

Hi Vladimir,

First, just a nit. I think that in LambdaFormEditor:

 289 private LambdaForm putInCache(Transform key, LambdaForm form) {
 290 key = key.withResult(form);
 291 for (int pass = 0; ; pass++) {
 292 Object c = lambdaForm.transformCache;
 293 if (c instanceof ConcurrentHashMap) {
 294 @SuppressWarnings(unchecked)
 295 ConcurrentHashMapTransform,Transform m = 
(ConcurrentHashMapTransform,Transform) c;
 296 Transform k = m.putIfAbsent(key, key);
 297 if (k == null) return form;
 298 LambdaForm result = k.get();
 299 if (result != null) {
 300 return result;
 301 } else {
 302 if (m.replace(key, k, key)) {
 303 return form;
 304 } else {
 305 continue;
 306 }
 307 }
 308 }
 309 assert(pass == 0);
 310 synchronized (lambdaForm) {
 311 c = lambdaForm.transformCache;
 312 if (c instanceof ConcurrentHashMap)
 313 continue;

...

 372 lambdaForm.transformCache = c = m;
 ^^^ put assignment to 'c' 
back in
 373 // The second iteration will update for this query, 
concurrently.
 374 continue;


...you could move the assignment to 'c' in line 292 out of for loop and 
put it back in line 372, since once 'c' is instance of CHM, 
lambdaForm.transformCache never changes again and if 'c' is not CHM yet, 
it is re-assigned in lines 311 and 372 before next loop.


Am I right?

Now what scares me (might be that I don't have an intimacy with 
LambdaForm class like you do). There is a situation where you publish 
LambdaForm instances via data race.


One form of LambdaForm.transformCache is an array of Transform objects 
(the other two forms are not problematic). Transform class has all 
fields final except the 'referent' field of SoftReference, which holds a 
LambdaForm instance. In the following line:


377 ta[idx] = key;


...you publish Transform object to an element of array with relaxed 
write, and in the following lines:


 271 } else {
 272 Transform[] ta = (Transform[])c;
 273 for (int i = 0; i  ta.length; i++) {
 274 Transform t = ta[i];
 275 if (t == null)  break;
 276 if (t.equals(key)) { k = t; break; }
 277 }
 278 }
 279 assert(k == null || key.equals(k));
 280 return (k != null) ? k.get() : null;


...you obtain the element of the array with no synchronization and a 
relaxed read and might return a non-null referent (the LambdaForm) which 
is then returned as an interned instance.


So can LambdaForm instances be published via data races without fear 
that they would appear half-initialized?


That's what I didn't know when I used a lazySet coupled with volatile 
get to access array elements in my version:


http://cr.openjdk.java.net/~plevart/misc/LambdaFormEditor.WeakCache/webrev.01/


Regards, Peter

On 12/03/2014 12:45 PM, Vladimir Ivanov wrote:

Aleksey, thanks for the review.

I haven't tried -XX:SoftRefLRUPolicyMSPerMB=0, but I did extensive 
testing on Octane/Nashorn with multiple low -Xmx levels + frequent 
Full GCs (8060147 [1] was the result of those experiments) and stress 
tested cache eviction with jdk/java/lang/invoke/LFCache tests in long 
running mode.


Best regards,
Vladimir Ivanov

[1] https://bugs.openjdk.java.net/browse/JDK-8060147

On 12/3/14, 3:11 PM, Aleksey Shipilev wrote:

On 12/01/2014 07:58 PM, Vladimir Ivanov wrote:

http://cr.openjdk.java.net/~vlivanov/8057020/webrev.00/
https://bugs.openjdk.java.net/browse/JDK-8057020


Looks okay, although the cache management logic gives me a headache
after the vacation. I thought I spotted a few bugs, but those were only
false positives.


The fix is to use SoftReferences to keep LambdaForms alive as long as
possible, but avoid throwing OOME until the caches are evicted. I
experimented with WeakReferences, but it doesn't hold LambdaForms for
long enough: LambdaForm cache hit rate degrades significantly and it
negatively affects application startup and warmup, since every
instantiated LambdaForm is precompiled to bytecode before usage.

Testing: jdk/java/lang/invoke/LFCache in stress mode + jck
(api/java_lang/invoke), jdk/java/lang/invoke, jdk/java/util/streams, 
octane


SoftReferences are tricky in the way they can get suddenly drop the
referent, and normal testing would not catch it (e.g. the normal
operation would reclaim softrefs under your feet almost never). Does
this code survive with -XX:SoftRefLRUPolicyMSPerMB=0?

Thanks,
-Aleksey.



___
mlvm-dev mailing list

Re: [9, 8u40] RFR (M): 8057020: LambdaForm caches should support eviction

[Peter Levart]

(Sorry for a re-send, forgot to include core-libs-dev)...

Hi Vladimir,

First, just a nit. I think that in LambdaFormEditor:

 289 private LambdaForm putInCache(Transform key, LambdaForm form) {
 290 key = key.withResult(form);
 291 for (int pass = 0; ; pass++) {
 292 Object c = lambdaForm.transformCache;
 293 if (c instanceof ConcurrentHashMap) {
 294 @SuppressWarnings(unchecked)
 295 ConcurrentHashMapTransform,Transform m = 
(ConcurrentHashMapTransform,Transform) c;
 296 Transform k = m.putIfAbsent(key, key);
 297 if (k == null) return form;
 298 LambdaForm result = k.get();
 299 if (result != null) {
 300 return result;
 301 } else {
 302 if (m.replace(key, k, key)) {
 303 return form;
 304 } else {
 305 continue;
 306 }
 307 }
 308 }
 309 assert(pass == 0);
 310 synchronized (lambdaForm) {
 311 c = lambdaForm.transformCache;
 312 if (c instanceof ConcurrentHashMap)
 313 continue;

...

 372 lambdaForm.transformCache = c = m;
 ^^^ put assignment to 'c' 
back in
 373 // The second iteration will update for this query, 
concurrently.
 374 continue;


...you could move the assignment to 'c' in line 292 out of for loop and 
put it back in line 372, since once 'c' is instance of CHM, 
lambdaForm.transformCache never changes again and if 'c' is not CHM yet, 
it is re-assigned in lines 311 and 372 before next loop.


Am I right?

Now what scares me (might be that I don't have an intimacy with 
LambdaForm class like you do). There is a situation where you publish 
LambdaForm instances via data race.


One form of LambdaForm.transformCache is an array of Transform objects 
(the other two forms are not problematic). Transform class has all 
fields final except the 'referent' field of SoftReference, which holds a 
LambdaForm instance. In the following line:


377 ta[idx] = key;


...you publish Transform object to an element of array with relaxed 
write, and in the following lines:


 271 } else {
 272 Transform[] ta = (Transform[])c;
 273 for (int i = 0; i  ta.length; i++) {
 274 Transform t = ta[i];
 275 if (t == null)  break;
 276 if (t.equals(key)) { k = t; break; }
 277 }
 278 }
 279 assert(k == null || key.equals(k));
 280 return (k != null) ? k.get() : null;


...you obtain the element of the array with no synchronization and a 
relaxed read and might return a non-null referent (the LambdaForm) which 
is then returned as an interned instance.


So can LambdaForm instances be published via data races without fear 
that they would appear half-initialized?


That's what I didn't know when I used a lazySet coupled with volatile 
get to access array elements in my version:


http://cr.openjdk.java.net/~plevart/misc/LambdaFormEditor.WeakCache/webrev.01/


Regards, Peter

On 12/03/2014 12:45 PM, Vladimir Ivanov wrote:

Aleksey, thanks for the review.

I haven't tried -XX:SoftRefLRUPolicyMSPerMB=0, but I did extensive 
testing on Octane/Nashorn with multiple low -Xmx levels + frequent 
Full GCs (8060147 [1] was the result of those experiments) and stress 
tested cache eviction with jdk/java/lang/invoke/LFCache tests in long 
running mode.


Best regards,
Vladimir Ivanov

[1] https://bugs.openjdk.java.net/browse/JDK-8060147

On 12/3/14, 3:11 PM, Aleksey Shipilev wrote:

On 12/01/2014 07:58 PM, Vladimir Ivanov wrote:

http://cr.openjdk.java.net/~vlivanov/8057020/webrev.00/
https://bugs.openjdk.java.net/browse/JDK-8057020


Looks okay, although the cache management logic gives me a headache
after the vacation. I thought I spotted a few bugs, but those were only
false positives.


The fix is to use SoftReferences to keep LambdaForms alive as long as
possible, but avoid throwing OOME until the caches are evicted. I
experimented with WeakReferences, but it doesn't hold LambdaForms for
long enough: LambdaForm cache hit rate degrades significantly and it
negatively affects application startup and warmup, since every
instantiated LambdaForm is precompiled to bytecode before usage.

Testing: jdk/java/lang/invoke/LFCache in stress mode + jck
(api/java_lang/invoke), jdk/java/lang/invoke, jdk/java/util/streams, 
octane


SoftReferences are tricky in the way they can get suddenly drop the
referent, and normal testing would not catch it (e.g. the normal
operation would reclaim softrefs under your feet almost never). Does
this code survive with -XX:SoftRefLRUPolicyMSPerMB=0?

Thanks,
-Aleksey.

Re: [9, 8u40] RFR (M): 8057020: LambdaForm caches should support eviction

[Peter Levart]

On 12/01/2014 05:58 PM, Vladimir Ivanov wrote:

http://cr.openjdk.java.net/~vlivanov/8057020/webrev.00/
https://bugs.openjdk.java.net/browse/JDK-8057020

There are 2 major LambdaForm caches: LambdaFormEditor-based and 
MethodTypeForm. The former is per-LambdaForm and the latter is per 
method type erased to basic types. The problem is that these caches 
don't support eviction, so they can hold LambdaForms forever.


Usually, it's not a problem since an application has very limited 
number of unique erased method types (e.g. on Octane/Nashorn it varies 
1,5-3k shapes).


The fix is to use SoftReferences to keep LambdaForms alive as long as 
possible, but avoid throwing OOME until the caches are evicted. I 
experimented with WeakReferences, but it doesn't hold LambdaForms for 
long enough: LambdaForm cache hit rate degrades significantly and it 
negatively affects application startup and warmup, since every 
instantiated LambdaForm is precompiled to bytecode before usage.


Testing: jdk/java/lang/invoke/LFCache in stress mode + jck 
(api/java_lang/invoke), jdk/java/lang/invoke, jdk/java/util/streams, 
octane


Thanks!

Best regards,
Vladimir Ivanov


Hi Vladimir,

So WeakReferences did not hold LambdaForms long enough even with strong 
back-reference from LambdaForm to the lambda form 'this' was derived 
from? So final derived LambdaForms (leaves) are not kept referenced from 
the code? Or did back-references keep intermediate LambdaForms in cache 
for too long (forever?) and you wanted them to be evicted too?


Regards, Peter

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: invoking a interface default method from within an InvocationHandler

[Peter Levart]

On 10/20/2014 11:25 AM, Cédric Champeau wrote:

For what it's worth, in Groovy, we have two separate pathes to implement
classes at runtime, that is to say generating proxies. In the case of an
interface, we rely on the JDK proxy just because it is *much* faster than
generating a class through ASM. I mean that generating a class in ASM is
orders of magnitude slower (at least in our case) than relying on the JDK
proxy adapters. Of course, it then depends on how often the method is
called, because at some point a generated class will be faster, but so far,
the choice we made is better in practice, because lots of people do not
realize that if they do foo as SomeInterface in a loop, they are
generating a new proxy for each call...


New proxy instance, not proxy class, right? j.l.r.Proxy API has a cache 
and only creates one Proxy class for a particular list of interfaces...


I agree that sun.misc.ProxyGenerator can be much faster at generating 
new class than ASM. But it's also more difficult to program/change. So 
it depend on what changes would be necessary...


Regards, Peter



2014-10-20 11:19 GMT+02:00 Paul Sandoz paul.san...@oracle.com:


On Oct 18, 2014, at 6:59 PM, Peter Levart peter.lev...@gmail.com wrote:

Hi Paul, Remi,

The complete solution will require new API, but the

java.lang.reflect.Proxy API could be improved a bit too. With introduction
of default interface methods it is somehow broken now. Default interface
methods enable interface evolution, but break existing Proxy
InvocationHandler(s) which are not prepared to handle new methods. So
perhaps a small improvement to Proxz API could be to add new factory method
that takes an additional a boolean flag indicating whether generated proxy
should override default methods and forward their invocation to
InvocationHandler or not. An all-or-nothing approach. I know that even that
is not trivial. Generated proxy class can implement multiple interfaces and
there can be conflicts. In that case, the easiest solution is to bail out
and refuse to generate a proxy class with such combination of interfaces.

It would also be useful to investigate a safe mechanism so that an
InvocationHandler implementation can call a default method, which then
allows an impl to explicitly deal with conflicts.



A starting point could be to re-implement current functionality using

ASM and then start experimenting with different approaches. What do you
think?
Not sure replacing the bytecode generating implementation, in
sun.misc.ProxyGenerator, with an ASM equivalent will really help all that
much functionality wise, that may be nice from a unification perspective
but does introduce some risk. A quicker route might be to work out how to
filter out the default methods that are overridden in the existing code.

Paul.

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev





___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: invoking a interface default method from within an InvocationHandler

[Peter Levart]

On 10/17/2014 11:58 AM, Paul Sandoz wrote:

On Oct 16, 2014, at 12:43 PM, Remi Forax fo...@univ-mlv.fr wrote:


On 10/15/2014 06:54 PM, Paul Sandoz wrote:

Hi Remi,

I did some brief evaluation of this area.

MethodHandleProxies.asInterfaceInstance currently does not support proxying to 
default methods. An InternalError will be thrown if a default method is 
invoked. It should be possible to fix using a trusted internal IMPL_LOOKUP to 
look up MHs that invokespecial. I believe this should be safe under the context 
of proxying.

Another solution is to do not rely on j.l.r.Proxy to implement 
asInterfaceInstance but to generate a proxy using ASM,
in that case, handling default methods is easy, just do nothing (i.e. do not 
override a default method).
It will also solve the fact that the proxies generated by asInterfaceInstance 
are currently super slow
because of the InvocationHandler API require boxing.

That's a good point, it's more involved and it would be nice to reuse 
implementing classes keyed off the functional interface, since a field can be 
defined for the proxying MH. Hmm... i wonder if we can leverage 
InnerClassLambdaMetafactory.



The real solution in my opinion is to create yet another API,

Yes, i am thinking just in the 9 time-frame, beyond that, as you indicate 
below, there is much more scope to improve this area (via invoke or a class 
dynamic).

Paul.


Hi Paul, Remi,

The complete solution will require new API, but the 
java.lang.reflect.Proxy API could be improved a bit too. With 
introduction of default interface methods it is somehow broken now. 
Default interface methods enable interface evolution, but break existing 
Proxy InvocationHandler(s) which are not prepared to handle new methods. 
So perhaps a small improvement to Proxz API could be to add new factory 
method that takes an additional a boolean flag indicating whether 
generated proxy should override default methods and forward their 
invocation to InvocationHandler or not. An all-or-nothing approach. I 
know that even that is not trivial. Generated proxy class can implement 
multiple interfaces and there can be conflicts. In that case, the 
easiest solution is to bail out and refuse to generate a proxy class 
with such combination of interfaces.


A starting point could be to re-implement current functionality using 
ASM and then start experimenting with different approaches. What do you 
think?


Regards, Peter


lets call it proxy 2.0 that takes an interface and a bootstrap method
as parameter and generate a class that will call the BSM when calling a method 
of the proxy class the first time
(the BSM will be called with a 4th parameter which is a method handle 
corresponding to the called method
so one can use MHs.reflectAs to get the corresponding j.l.r.Method object if 
necessary).
For a default method, the 4th parameter will be a method handle created with 
findSuper, so a user can choose
to implement it or to delegate to this mh.
And asInterfaceInstance is just a foldArguments between the method handle taken 
as parameter and
an exactInvoker (or a genericInvoker if parameter types mismatch).

Compared to a j.l.r.Proxy, the proxy 2.0 API also need a way to store fields 
(or at least values) inside the generated proxy class
and a way to query that fields inside the BSM. This can be done exactly like 
this is done by the lambda proxy,
instead of returning a proxy class, the proxy 2.0 API will return a method 
handle that acts as a factory for creating
proxy instances of the proxy class (if the factory takes no arguments, as for 
the lambda proxy, the same constant object can be returned).
To get a mh getter from inside the BSM, because the BSM already have the lookup 
object, all you need is a convention that says
that the field names are something like arg0, arg1, etc.

The nice thing about this API is that it cleanly separate the information that 
can be process from proxied interface(s)
(by example, JavaEE annotations) and the ones, more dynamic, that are specific 
to a proxy instance.
It also removes one level of indirection compared to the InvocationHandler 
proxy.



___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: [9] RFR (S): 8058892: FILL_ARRAYS and ARRAYS are eagely initialized in MethodHandleImpl

[Peter Levart]

On 10/02/2014 06:55 PM, Vladimir Ivanov wrote:

Small update:
http://cr.openjdk.java.net/~vlivanov/8058892/webrev.01/

Need to reorder initialization sequence in MHI.Lazy. Initialized 
FILL_ARRAYS and ARRAYS are required for later MH lookups.


Additional testing:
  * jck (api/java_lang/invoke)
  * jdk/java/lang/invoke, jdk/java/util/streams w/ -ea -esa and 
COMPILE_THRESHOLD={0,30}


Best regards,
Vladimir Ivanov


Hi Vladimir,

I have a comment that does not directly pertain to the code changes (the 
initialization of arrays) but to the sub-optimal implementation of 
fillArray methods I noticed by the way. While it is nice to use 
varargs makeArray helper method with array methods to construct the 
array, the same strategy used with fillWithArguments in fillArray 
methods makes a redundant array that is then copied to target array and 
discarded. The redundant copying has a price. Here's a benchmark 
(Aleksey, please bear with me):


@State(Scope.Benchmark)
public class FillArrayTest {

private Object
a0 = new Object(),
a1 = new Object(),
a2 = new Object(),
a3 = new Object(),
a4 = new Object(),
a5 = new Object(),
a6 = new Object(),
a7 = new Object();


private static void fillWithArguments(Object[] a, int pos, 
Object... args) {

System.arraycopy(args, 0, a, pos, args.length);
}

private static Object[] fillArray(
Integer pos, Object[] a,
Object a0, Object a1, Object a2, Object a3,
Object a4, Object a5, Object a6, Object a7
) {
fillWithArguments(a, pos, a0, a1, a2, a3, a4, a5, a6, a7);
return a;
}

private static Object[] fillArrayAlt(
Integer pos, Object[] a,
Object a0, Object a1, Object a2, Object a3,
Object a4, Object a5, Object a6, Object a7
) {
int i = pos;
a[i++] = a0;
a[i++] = a1;
a[i++] = a2;
a[i++] = a3;
a[i++] = a4;
a[i++] = a5;
a[i++] = a6;
a[i++] = a7;
return a;
}

@Benchmark
public Object[] fillArray() {
return fillArray(0, new Object[8], a0, a1, a2, a3, a4, a5, a6, a7);
}

@Benchmark
public Object[] fillArrayAlt() {
return fillArrayAlt(0, new Object[8], a0, a1, a2, a3, a4, a5, 
a6, a7);

}
}


The results on my i7 with JMH arguments -i 8 -wi 5 -f 1 -gc true:

Benchmark  Mode   SamplesScore Score 
errorUnits
j.t.FillArrayTest.fillArray   thrpt 8 48601447.674 
5414853.634ops/s
j.t.FillArrayTest.fillArrayAltthrpt 8 90044973.732 
8713725.735ops/s



So fillArrayAlt is nearly twice as fast...

Regards, Peter




On 10/2/14, 7:52 PM, Vladimir Ivanov wrote:

http://cr.openjdk.java.net/~vlivanov/8058892/webrev.00/
https://bugs.openjdk.java.net/browse/JDK-8058892

Core j.l.i classes are preloaded during VM startup in order to avoid
possible deadlock when accessing JSR292-related functionality from
multiple threads. After LF sharing-related changes, FILL_ARRAYS and
ARRAYS are initialized too early. It affects startup time  footprint of
applications that don't use JSR292.

The fix is to move these fields into MHI.Lazy class, thus delaying their
initialization to the first usage of JSR292 API.

Testing: failing test, manual (measured HelloWorld app startup time;
compared -XX:+PrintCompilation logs)

Best regards,
Vladimir Ivanov

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: [9] RFR (S): 8058892: FILL_ARRAYS and ARRAYS are eagely initialized in MethodHandleImpl

[Peter Levart]

On 10/02/2014 08:42 PM, Vitaly Davidovich wrote:


AFIK, varargs (up to something like 64 args) should be eliminated by 
EA.  Peter, can you add another jmh test that uses varargs but doesn't 
call into System.arraycopy but uses the hand rolled version like your 
at method? I'm wondering if that makes EA not kick in.




Hm, here is a modified benchmark (I also eliminated repeatable 
allocation of target array):



@State(Scope.Benchmark)
public class FillArrayTest {

private Object
a0 = new Object(),
a1 = new Object(),
a2 = new Object(),
a3 = new Object(),
a4 = new Object(),
a5 = new Object(),
a6 = new Object(),
a7 = new Object();


private Object[] target = new Object[8];

private static void fillWithArguments(Object[] a, int pos, 
Object... args) {

System.arraycopy(args, 0, a, pos, args.length);
}

private static Object[] fillArray(
Integer pos, Object[] a,
Object a0, Object a1, Object a2, Object a3,
Object a4, Object a5, Object a6, Object a7
) {
fillWithArguments(a, pos, a0, a1, a2, a3, a4, a5, a6, a7);
return a;
}

private static void fillWithArgumentsCopyLoop(Object[] a, int pos, 
Object... args) {

for (int i = 0; i  args.length; i++) {
a[i + pos] = args[i];
}
}

private static Object[] fillArrayCopyLoop(
Integer pos, Object[] a,
Object a0, Object a1, Object a2, Object a3,
Object a4, Object a5, Object a6, Object a7
) {
fillWithArgumentsCopyLoop(a, pos, a0, a1, a2, a3, a4, a5, a6, a7);
return a;
}

private static Object[] fillArrayAlt(
Integer pos, Object[] a,
Object a0, Object a1, Object a2, Object a3,
Object a4, Object a5, Object a6, Object a7
) {
int i = pos;
a[i++] = a0;
a[i++] = a1;
a[i++] = a2;
a[i++] = a3;
a[i++] = a4;
a[i++] = a5;
a[i++] = a6;
a[i++] = a7;
return a;
}

@Benchmark
public Object[] fillArray() {
return fillArray(0, target, a0, a1, a2, a3, a4, a5, a6, a7);
}

@Benchmark
public Object[] fillCopyLoop() {
return fillArrayCopyLoop(0, target, a0, a1, a2, a3, a4, a5, a6, 
a7);

}

@Benchmark
public Object[] fillArrayAlt() {
return fillArrayAlt(0, target, a0, a1, a2, a3, a4, a5, a6, a7);
}
}



The results:


Benchmark  Mode   SamplesScore Score 
errorUnits
j.t.FillArrayTest.fillArray   thrpt 8 76534019.978 
3063590.310ops/s
j.t.FillArrayTest.fillArrayAltthrpt 8 141640280.270 
7815152.038ops/s
j.t.FillArrayTest.fillCopyLoopthrpt 8 82050640.406 
4055652.247ops/s



The fillCopyLoop seems a little faster. I don't know if this is because 
of possible elimination of allocation. The fillArrayAlt is still almost 
2x as fast.


Peter


Sent from my phone

On Oct 2, 2014 2:34 PM, Peter Levart peter.lev...@gmail.com 
mailto:peter.lev...@gmail.com wrote:



On 10/02/2014 06:55 PM, Vladimir Ivanov wrote:

Small update:
http://cr.openjdk.java.net/~vlivanov/8058892/webrev.01/
http://cr.openjdk.java.net/%7Evlivanov/8058892/webrev.01/

Need to reorder initialization sequence in MHI.Lazy.
Initialized FILL_ARRAYS and ARRAYS are required for later MH
lookups.

Additional testing:
  * jck (api/java_lang/invoke)
  * jdk/java/lang/invoke, jdk/java/util/streams w/ -ea -esa
and COMPILE_THRESHOLD={0,30}

Best regards,
Vladimir Ivanov


Hi Vladimir,

I have a comment that does not directly pertain to the code
changes (the initialization of arrays) but to the sub-optimal
implementation of fillArray methods I noticed by the way. While
it is nice to use varargs makeArray helper method with array
methods to construct the array, the same strategy used with
fillWithArguments in fillArray methods makes a redundant array
that is then copied to target array and discarded. The redundant
copying has a price. Here's a benchmark (Aleksey, please bear with
me):

@State(Scope.Benchmark)
public class FillArrayTest {

private Object
a0 = new Object(),
a1 = new Object(),
a2 = new Object(),
a3 = new Object(),
a4 = new Object(),
a5 = new Object(),
a6 = new Object(),
a7 = new Object();


private static void fillWithArguments(Object[] a, int pos,
Object... args) {
System.arraycopy(args, 0, a, pos, args.length);
}

private static Object[] fillArray(
Integer pos, Object[] a,
Object a0, Object a1, Object a2, Object a3,
Object a4, Object a5, Object a6, Object a7
) {
fillWithArguments(a, pos, a0, a1, a2, a3, a4

Re: [9] RFR (S): 8058892: FILL_ARRAYS and ARRAYS are eagely initialized in MethodHandleImpl

[Peter Levart]

On 10/02/2014 09:28 PM, Vitaly Davidovich wrote:
Yes, this is the version I wanted to see (i.e. where the only diff is 
varargs, and not any anything else like array iteration, etc).
Why did fillCopyLoop() get some much worse than in your prior email 
though?


82050640.406  (+-4055652.247) vs. 82395900.795  (+-2794747.540)

I don't see a noticeable difference (there's a lot of jitter - didn't 
turn off tiered compilation).


Regards, Peter





On Thu, Oct 2, 2014 at 3:12 PM, Peter Levart peter.lev...@gmail.com 
mailto:peter.lev...@gmail.com wrote:


Well, if I add the following variant to the mix:

private static void fillWith8Arguments(Object[] a, int pos,
Object... args) {
a[pos] = args[0];
a[pos + 1] = args[1];
a[pos + 2] = args[2];
a[pos + 3] = args[3];
a[pos + 4] = args[4];
a[pos + 5] = args[5];
a[pos + 6] = args[6];
a[pos + 7] = args[7];
}

private static Object[] fillArrayByHand(
Integer pos, Object[] a,
Object a0, Object a1, Object a2, Object a3,
Object a4, Object a5, Object a6, Object a7
) {
fillWith8Arguments(a, pos, a0, a1, a2, a3, a4, a5, a6, a7);
return a;
}

@Benchmark
public Object[] fillByHand() {
return fillArrayByHand(0, target, a0, a1, a2, a3, a4, a5,
a6, a7);
}

The results:

Benchmark  Mode   Samples Score  Score
errorUnits
j.t.FillArrayTest.fillArray   thrpt 8 75994667.408 
4169836.951ops/s
j.t.FillArrayTest.fillArrayAltthrpt 8 142761145.565 
7127589.095ops/s
j.t.FillArrayTest.fillByHand  thrpt 8 141206898.861 
6435932.932ops/s
j.t.FillArrayTest.fillCopyLoopthrpt 8 82395900.795 
2794747.540ops/s



...show that with fillByHand, varargs array is indeed eliminated.
But then the helper method is not of any help, since it's not
reusable for different array lengths...


Regards, Peter


On 10/02/2014 08:54 PM, Peter Levart wrote:


On 10/02/2014 08:42 PM, Vitaly Davidovich wrote:


AFIK, varargs (up to something like 64 args) should be
eliminated by EA.  Peter, can you add another jmh test that uses
varargs but doesn't call into System.arraycopy but uses the hand
rolled version like your at method? I'm wondering if that makes
EA not kick in.



Hm, here is a modified benchmark (I also eliminated repeatable
allocation of target array):


@State(Scope.Benchmark)
public class FillArrayTest {

private Object
a0 = new Object(),
a1 = new Object(),
a2 = new Object(),
a3 = new Object(),
a4 = new Object(),
a5 = new Object(),
a6 = new Object(),
a7 = new Object();


private Object[] target = new Object[8];

private static void fillWithArguments(Object[] a, int pos,
Object... args) {
System.arraycopy(args, 0, a, pos, args.length);
}

private static Object[] fillArray(
Integer pos, Object[] a,
Object a0, Object a1, Object a2, Object a3,
Object a4, Object a5, Object a6, Object a7
) {
fillWithArguments(a, pos, a0, a1, a2, a3, a4, a5, a6, a7);
return a;
}

private static void fillWithArgumentsCopyLoop(Object[] a, int
pos, Object... args) {
for (int i = 0; i  args.length; i++) {
a[i + pos] = args[i];
}
}

private static Object[] fillArrayCopyLoop(
Integer pos, Object[] a,
Object a0, Object a1, Object a2, Object a3,
Object a4, Object a5, Object a6, Object a7
) {
fillWithArgumentsCopyLoop(a, pos, a0, a1, a2, a3, a4, a5,
a6, a7);
return a;
}

private static Object[] fillArrayAlt(
Integer pos, Object[] a,
Object a0, Object a1, Object a2, Object a3,
Object a4, Object a5, Object a6, Object a7
) {
int i = pos;
a[i++] = a0;
a[i++] = a1;
a[i++] = a2;
a[i++] = a3;
a[i++] = a4;
a[i++] = a5;
a[i++] = a6;
a[i++] = a7;
return a;
}

@Benchmark
public Object[] fillArray() {
return fillArray(0, target, a0, a1, a2, a3, a4, a5, a6, a7);
}

@Benchmark
public Object[] fillCopyLoop() {
return fillArrayCopyLoop(0, target, a0, a1, a2, a3, a4,
a5, a6, a7);
}

@Benchmark
public Object[] fillArrayAlt() {
return fillArrayAlt(0, target, a0, a1, a2, a3, a4, a5,
a6, a7);
}
}



The results:


Benchmark

Re: [9] RFR (L): 8057042: LambdaFormEditor: derive new LFs from a base LF

[Peter Levart]

On 09/03/2014 03:25 PM, Vladimir Ivanov wrote:

Peter,

Thanks for the feedback.

 In LambdaFormEditor, where Transform[] is promoted into

ConcurrentHashMapTransform, Transform:

  339 ConcurrentHashMapTransform, Transform m =
new ConcurrentHashMap(MAX_CACHE_ARRAY_SIZE * 2);
  340 for (Transform k : ta) {
  341 if (k != null)  continue;
  342 m.put(k, k);
  343 }
  344 lambdaForm.transformCache = m;
  345 // The second iteration will update for this
query, concurrently.
  346 continue;


I think line 341 is wrong. It should be:

 if (k == null) break;

shouldn't it?

Good catch! Fixed.



I think it can even be removed or replaced with something like:

assert k != null;

...since null entry in array is not possible in this situation - 
promotion to CHM happens only when array is full.



Regards, Peter


Best regards,
Vladimir Ivanov




Regards, Peter




On 9/2/14, 5:57 PM, Vladimir Ivanov wrote:

https://bugs.openjdk.java.net/browse/JDK-8057042

LambdaFormEditor provides an API to transform LambdaForms. Deriving 
new

LambdaForms from a base one allows to cache and reuse results of
repeated transformations.

BMH binding is rewritten to use LambdaFormEditor.

Testing: jdk/java/lang/invoke, jdk/java/util/streams, nashorn, 
octane w/

-ea -esa and COMPILE_THRESHOLD={0,30}.

Reviewed-by: vlivanov, ?
Contributed-by: john.r.r...@oracle.com

Thanks!

Best regards,
Vladimir Ivanov
___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: [9] RFR (L): 8057042: LambdaFormEditor: derive new LFs from a base LF

[Peter Levart]

On 09/03/2014 03:29 PM, Vladimir Ivanov wrote:

Peter,

Yes, it's a known problem [1].
There are other caches (in MethodTypeForm, for example), which 
shouldn't retain their elements.


Hi Vladimir,

I was tempted to see what it would take to use weakly referenced 
LambdaForms from cache entries (LambdaFormEditor.Transform objects).


This is what I came up with:

http://cr.openjdk.java.net/~plevart/misc/LambdaFormEditor.WeakCache/webrev.01/

In this modification on top of your patch, a reference to LambdaForm 
from Transform is not a final field any more (WeakReference has a normal 
field), so I had to change LambdaForm.transformCacheto be volatile field 
to enable safe publication of Transform objects and transiently 
LambdaForm objects. I also used ordered writes with volatile reads for 
Transform[] array elements where necessary. CHM already does what's 
necessary. If LambdaForm objects are unsafe-publication-tolerable, this 
can be simplified.


I have made a little effort to re-use slots occupied by cleared 
Transform objects, but nothing fancy like using ReferenceQueue(s) or 
such, since there would have to be a queue per LambdaForm and this would 
bring some overhead with it. Transform objects are very compact (even 
when they extend a WeakReference) and majority of heap is released by 
free-ing weakly reachable LambdaForm objects which can be quite big and 
deep sometimes.


A cache forms a tree of LambdaForm objects. Child LFs are derived from 
base (parent) LFs. parent - child references are weak, but I added 
child - parent references which are strong. If there is a strong 
reference to some cached LF from the application, the whole path leading 
from the root LF to the cached LF is kept alive this way, so that any 
code that wishes to follow that path can get to the cached LF.


So do you think that cached LambdaForm(s) are so general that they are 
better cached for the life of JVM even in long-running application 
servers that re-deploy apps on the fly and using WeakReference(s) is not 
necessary?


Regards, Peter



Best regards,
Vladimir Ivanov

[1] https://bugs.openjdk.java.net/browse/JDK-8057020

On 9/3/14, 3:43 PM, Peter Levart wrote:

Hi Vladimir,

I'm sure you and John have thought about it through, but I'll ask
anyway. Are cached LambdaForms going to stay around forever? What about
using a WeakReferenceLambdaForm (LambdaFormEditor.Transform could
extend WeakReference). This way unused LambdaForms would get GCed.

Regards, Peter

On 09/02/2014 03:59 PM, Vladimir Ivanov wrote:

Webrev: http://cr.openjdk.java.net/~vlivanov/8057042/webrev.00

Best regards,
Vladimir Ivanov

On 9/2/14, 5:57 PM, Vladimir Ivanov wrote:

https://bugs.openjdk.java.net/browse/JDK-8057042

LambdaFormEditor provides an API to transform LambdaForms. Deriving 
new

LambdaForms from a base one allows to cache and reuse results of
repeated transformations.

BMH binding is rewritten to use LambdaFormEditor.

Testing: jdk/java/lang/invoke, jdk/java/util/streams, nashorn, 
octane w/

-ea -esa and COMPILE_THRESHOLD={0,30}.

Reviewed-by: vlivanov, ?
Contributed-by: john.r.r...@oracle.com

Thanks!

Best regards,
Vladimir Ivanov
___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev




___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: [9] RFR (L): 8057042: LambdaFormEditor: derive new LFs from a base LF

[Peter Levart]

On 09/02/2014 03:59 PM, Vladimir Ivanov wrote:

Webrev: http://cr.openjdk.java.net/~vlivanov/8057042/webrev.00

Best regards,
Vladimir Ivanov


Hi Vladimir,

In LambdaFormEditor, where Transform[] is promoted into 
ConcurrentHashMapTransform, Transform:


 339 ConcurrentHashMapTransform, Transform m = new 
ConcurrentHashMap(MAX_CACHE_ARRAY_SIZE * 2);
 340 for (Transform k : ta) {
 341 if (k != null)  continue;
 342 m.put(k, k);
 343 }
 344 lambdaForm.transformCache = m;
 345 // The second iteration will update for this query, 
concurrently.
 346 continue;


I think line 341 is wrong. It should be:

if (k == null) break;

shouldn't it?



Regards, Peter




On 9/2/14, 5:57 PM, Vladimir Ivanov wrote:

https://bugs.openjdk.java.net/browse/JDK-8057042

LambdaFormEditor provides an API to transform LambdaForms. Deriving new
LambdaForms from a base one allows to cache and reuse results of
repeated transformations.

BMH binding is rewritten to use LambdaFormEditor.

Testing: jdk/java/lang/invoke, jdk/java/util/streams, nashorn, octane w/
-ea -esa and COMPILE_THRESHOLD={0,30}.

Reviewed-by: vlivanov, ?
Contributed-by: john.r.r...@oracle.com

Thanks!

Best regards,
Vladimir Ivanov
___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: [9] RFR (L): 8057042: LambdaFormEditor: derive new LFs from a base LF

[Peter Levart]

Hi Vladimir,

I'm sure you and John have thought about it through, but I'll ask 
anyway. Are cached LambdaForms going to stay around forever? What about 
using a WeakReferenceLambdaForm (LambdaFormEditor.Transform could 
extend WeakReference). This way unused LambdaForms would get GCed.


Regards, Peter

On 09/02/2014 03:59 PM, Vladimir Ivanov wrote:

Webrev: http://cr.openjdk.java.net/~vlivanov/8057042/webrev.00

Best regards,
Vladimir Ivanov

On 9/2/14, 5:57 PM, Vladimir Ivanov wrote:

https://bugs.openjdk.java.net/browse/JDK-8057042

LambdaFormEditor provides an API to transform LambdaForms. Deriving new
LambdaForms from a base one allows to cache and reuse results of
repeated transformations.

BMH binding is rewritten to use LambdaFormEditor.

Testing: jdk/java/lang/invoke, jdk/java/util/streams, nashorn, octane w/
-ea -esa and COMPILE_THRESHOLD={0,30}.

Reviewed-by: vlivanov, ?
Contributed-by: john.r.r...@oracle.com

Thanks!

Best regards,
Vladimir Ivanov
___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: RFR: 8000975: (process) Merge UNIXProcess.java.bsd UNIXProcess.java.linux ( .solaris .aix)

[Peter Levart]

This is a ping for any Reviewer and also a question for Vladimir.

Hello Vladimir,

What do you think about the classloader issue in the resolving of 
classes in MemberName.getMethodType() described below?


Regards, Peter

On 04/23/2014 04:21 PM, Paul Sandoz wrote:

Hi Peter,

IMHO such security manager usage by the test is v. fragile and we should try 
and find a safer alternative if possible.

However, there may also be an issue with lambda form code. (About a month ago i 
too was looking, internally, at this kind of issue and thought there was a 
questionable use of the application/system class loader when initializing the 
LambdaForm class, but then i got distracted with other stuff and forgot about 
it.)

Your one-liner fix seems to do the trick, but I think we need Vladimir to 
confirm this is OK.

Paul.

On Apr 17, 2014, at 11:49 PM, Peter Levart peter.lev...@gmail.com wrote:


Hi,

I'm cross-posting this on the mlvm-dev mailing list, because I think it 
concerns internal MHs implementation.

While replacing some inner classes with lambdas in java.lang.UNIXProcess class, a jtreg 
test started failing. This test is employing a security manager with an unusual 
configuration. It defines java.util as a package for which access should be 
checked against the set of permissions. The class initialization of UNIXProcess class 
initializes some lambdas and their initialization fails with the following stack-trace:


java.lang.ExceptionInInitializerError
at 
java.lang.invoke.DirectMethodHandle.makePreparedLambdaForm(DirectMethodHandle.java:256)
at 
java.lang.invoke.DirectMethodHandle.preparedLambdaForm(DirectMethodHandle.java:221)
at 
java.lang.invoke.DirectMethodHandle.preparedLambdaForm(DirectMethodHandle.java:210)
at java.lang.invoke.DirectMethodHandle.make(DirectMethodHandle.java:82)
at 
java.lang.invoke.MethodHandles$Lookup.getDirectMethodCommon(MethodHandles.java:1638)
at 
java.lang.invoke.MethodHandles$Lookup.getDirectMethodNoSecurityManager(MethodHandles.java:1602)
at 
java.lang.invoke.MethodHandles$Lookup.getDirectMethodForConstant(MethodHandles.java:1778)
at 
java.lang.invoke.MethodHandles$Lookup.linkMethodHandleConstant(MethodHandles.java:1727)
at 
java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(MethodHandleNatives.java:442)
at java.lang.UNIXProcess$Platform.get(UNIXProcess.java:147)
at java.lang.UNIXProcess.clinit(UNIXProcess.java:160)
at java.lang.ProcessImpl.start(ProcessImpl.java:130)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1023)
at java.lang.Runtime.exec(Runtime.java:620)
at java.lang.Runtime.exec(Runtime.java:485)
at SecurityManagerClinit.main(SecurityManagerClinit.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:484)
at 
com.sun.javatest.regtest.MainWrapper$MainThread.run(MainWrapper.java:94)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.security.AccessControlException: access denied 
(java.lang.RuntimePermission accessClassInPackage.java.util)
at 
java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
at 
java.security.AccessController.checkPermission(AccessController.java:884)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:541)
at 
java.lang.SecurityManager.checkPackageAccess(SecurityManager.java:1481)
*at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:305)*
at java.lang.ClassLoader.loadClass(ClassLoader.java:359)
at 
sun.invoke.util.BytecodeDescriptor.parseSig(BytecodeDescriptor.java:83)
*at 
sun.invoke.util.BytecodeDescriptor.parseMethod(BytecodeDescriptor.java:54)*
at 
sun.invoke.util.BytecodeDescriptor.parseMethod(BytecodeDescriptor.java:41)
at 
java.lang.invoke.MethodType.fromMethodDescriptorString(MethodType.java:911)
at java.lang.invoke.MemberName.getMethodType(MemberName.java:144)
at 
java.lang.invoke.LambdaForm.computeInitialPreparedForms(LambdaForm.java:477)
at java.lang.invoke.LambdaForm.clinit(LambdaForm.java:1641)


I think I found the root cause of the problem. It's nothing wrong with the test (although making 
java.util as an access-checked package is a little unusual). The problem, I think, is 
in MH's LambdaForm implementation. Although the UNIXProcess class is a system class (loaded by 
bootstrap class loader), MHs created by evaluating lambda expressions in this class, trigger 
loading a class in java.util package using AppClassLoader as the initiating class 
loader, which involves package access checks. This should not happen, I think, if only the system

Re: RFR: 8000975: (process) Merge UNIXProcess.java.bsd UNIXProcess.java.linux ( .solaris .aix)

[Peter Levart]

 on it. The 
ClassLoader.getSystemClassLoader() is the application class-loader 
(AppClassLoader).


I think the right thing to do would be to use bootstrap class loader if 
the ClassLoader parameter passed to *BytecodeDescriptor.parseMethod *was 
null. The fix would be as follows:



===
--- jdk/src/share/classes/sun/invoke/util/BytecodeDescriptor.java 
(revision 9770:8371276d52c0)
+++ jdk/src/share/classes/sun/invoke/util/BytecodeDescriptor.java 
(revision 9770+:8371276d52c0+)

@@ -43,8 +43,6 @@

 static ListClass? parseMethod(String bytecodeSignature,
 int start, int end, ClassLoader loader) {
-if (loader == null)
-loader = ClassLoader.getSystemClassLoader();
 String str = bytecodeSignature;
 int[] i = {start};
 ArrayListClass? ptypes = new ArrayListClass?();
@@ -80,7 +78,7 @@
 i[0] = endc+1;
 String name = str.substring(begc, endc).replace('/', '.');
 try {
-return loader.loadClass(name);
+return Class.forName(name, false, loader);
 } catch (ClassNotFoundException ex) {
 throw new TypeNotPresentException(name, ex);
 }



What do you think? Am I right?


Regards, Peter

Regards, Peter
On 04/17/2014 06:50 PM, Martin Buchholz wrote:
This test tries to make sure we have all of the doPrivileged blocks in 
place.  Which is hard to do.  I don't recall the details.  It would 
seem better to simply test that processes can be executed if the very 
specific permission to do just that was provided.  Which IIRC 
ProcessBuilder/Basic.java already has tests for.  Ohhh but to test 
clinit behavior, they have to be in a new JVM instance, so the 
existing tests in Basic are insufficient for that purpose.  But it 
would be cumbersome to transplant current security tests from Basic 
into an environment where it's the only invocation of the process api.



On Thu, Apr 17, 2014 at 8:33 AM, Peter Levart peter.lev...@gmail.com 
mailto:peter.lev...@gmail.com wrote:


Hi Martin,

Since you are the author of the
test/java/lang/ProcessBuilder/SecurityManagerClinit.java test, I
thought I'll ask you about the purpose of the following line in
the test just before setting up the Policy and SecurityManager:

// A funky contrived security setup, just for bug repro
purposes.
java.security.Security.setProperty(package.access,
java.util);

The merge of UNIXProcess.java.* sources fails this test since it
now uses lambdas in various places and it seems that MHs are
involved that need access to java.util package from a class loaded
by AppClassLoader. Would it defeat the purpose of the test if
java.util package was replaced by say sun.misc which is
normally restricted?

Regards, Peter


On 04/04/2014 09:24 PM, roger riggs wrote:

Hi Peter,

I ran into one test problem when running this through its paces.

The test: test/java/lang/ProcessBuilder/SecurityManagerClinit.java
throws a package access exception while creating a lambda
due to the wacky security manager and forbidding of access to
java.util.
I hacked the test to remove the limitation on java.util.

This looks good to me but a more experienced Reviewer should
look at it.

Thanks, Roger


Right,

Here it is:

http://cr.openjdk.java.net/~plevart/jdk9-dev/UNIXProcess/webrev.06/

http://cr.openjdk.java.net/%7Eplevart/jdk9-dev/UNIXProcess/webrev.06/



Stack dump from test:
java.lang.ExceptionInInitializerError
at

java.lang.invoke.DirectMethodHandle.makePreparedLambdaForm(DirectMethodHandle.java:256)
at

java.lang.invoke.DirectMethodHandle.preparedLambdaForm(DirectMethodHandle.java:221)
at

java.lang.invoke.DirectMethodHandle.preparedLambdaForm(DirectMethodHandle.java:210)
at
java.lang.invoke.DirectMethodHandle.make(DirectMethodHandle.java:82)
at

java.lang.invoke.MethodHandles$Lookup.getDirectMethodCommon(MethodHandles.java:1638)
at

java.lang.invoke.MethodHandles$Lookup.getDirectMethodNoSecurityManager(MethodHandles.java:1602)
at

java.lang.invoke.MethodHandles$Lookup.getDirectMethodForConstant(MethodHandles.java:1778)
at

java.lang.invoke.MethodHandles$Lookup.linkMethodHandleConstant(MethodHandles.java:1727)
at

java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(MethodHandleNatives.java:442)
at
java.lang.UNIXProcess$Platform.get(UNIXProcess.java:147)
at java.lang.UNIXProcess.clinit(UNIXProcess.java:160)
at java.lang.ProcessImpl.start(ProcessImpl.java:130

Re: [9] RFR (S): 8036117: MethodHandles.catchException doesn't handle VarargsCollector right (8034120 failed)

[Peter Levart]

On 03/11/2014 05:46 PM, Vladimir Ivanov wrote:

So all you're achieving by annotating prepend() method is that any
exception stack trace, in case it is thrown inside the prepend() method,
will hide where it was thrown from. In case all LambdaForm frames
leading to the prepend() method were also hidden, the exception would
appear to be thrown from the invocation of the MH...

Yes. Also, Thread.dumpStack() won't list it.
Do you see any problem here?

It depends on what you're trying to achieve. Whether you're trying to 
hide the internals even in situations where something goes wrong or 
you're trying to simplify debugging in case something goes wrong. But I 
guess not much can go wrong in this method as I suspect the passed-in 
'array' can't be null, so it really doesn't matter.


Regards, Peter


Best regards,
Vladimir Ivanov



Regards, Peter



Best regards,
Vladimir Ivanov


Regards, Peter



There's not much value in it in this particular case, but I 
decided to

reduce possible noise.

Best regards,
Vladimir Ivanov

On 3/11/14 3:35 PM, Peter Levart wrote:

Hi,

Excuse my ignorant question: What is the purpose of 
@LambdaForm.Hidden

annotation?

I suspect it has to do with hiding the call frames in stack traces
that
are part of LambdaForm invocation chain. In this case, method:

 private static Object[] prepend(Object elem, Object[] array)

in MethodHandleImpl need not be annotated with this annotation, 
since

it's call frame is not on stack when one of the target methods is
executed. It's just a function used to calculate the argument of the
call. In fact, if prepend() ever throws exception (NPE in case
array is
null?), It would be preferable that it's call frame is visible in 
the

stack trace.

Am I right or am I just talking nonsense?

Regards, Peter


On 03/11/2014 12:12 AM, Vladimir Ivanov wrote:

John, Chris, thanks!

Best regards,
Vladimir Ivanov

On 3/11/14 3:08 AM, Christian Thalinger wrote:

Even better.

On Mar 10, 2014, at 3:21 PM, Vladimir Ivanov
vladimir.x.iva...@oracle.com wrote:


Chris, thanks for the review.

John suggested an elegant way to fix the problem - use
asFixedArity.

Updated fix:
http://cr.openjdk.java.net/~vlivanov/8036117/webrev.01/

Best regards,
Vladimir Ivanov

On 3/8/14 4:51 AM, Christian Thalinger wrote:
Seems good to me.  I’d like to have another name for this 
method:


+ private static Object invokeCustom(MethodHandle target,
Object... args) throws Throwable {

On Mar 4, 2014, at 12:00 PM, Vladimir Ivanov
vladimir.x.iva...@oracle.com wrote:


http://cr.openjdk.java.net/~vlivanov/8036117/webrev.00/
https://bugs.openjdk.java.net/browse/JDK-8036117
84 lines changed: 74 ins; 3 del; 7 mod

I have to revert a cleanup I did for 8027827.
MethodHandle.invokeWithArguments (and generic invocation) has
unpleasant
peculiarity in behavior when used with VarargsCollector. So,
unfortunately, invokeWithArguments is not an option there.

Looking at the API (excerpts from javadoc [1] [2]), the 
following

condition doesn't hold in that case:
   trailing parameter type of the caller is a reference type
identical
to or assignable to the trailing parameter type of the 
adapter.


Example:
   target.invokeWithArguments((Object[])args)
   =
target.invoke((Object)o1,(Object)o2,(Object)o3)
   =/
   target.invokeExact((Object)o1, (Object)o2, (Object[])o3)

because Object !: Object[].

The fix is to skip unnecessary conversion when invoking a 
method

handle
and just do a pairwise type conversion.

Testing: failing test case, nashorn w/ experimental features
(octane)

Thanks!

Best regards,
Vladimir Ivanov

[1] MethodHandle.invokeWithArguments
Performs a variable arity invocation, ..., as if via an 
inexact

invoke
from a call site which mentions only the type Object, and whose
arity is
the length of the argument array.

[2] MethodHandle.asVarargsCollector
When called with plain, inexact invoke, if the caller type is
the
same
as the adapter, the adapter invokes the target as with
invokeExact.
(This is the normal behavior for invoke when types match.)

Otherwise, if the caller and adapter arity are the same, and 
the

trailing parameter type of the caller is a reference type
identical to
or assignable to the trailing parameter type of the adapter, 
the

arguments and return values are converted pairwise, as if by
asType on a
fixed arity method handle.

Otherwise, the arities differ, or the adapter's trailing
parameter
type
is not assignable from the corresponding caller type. In this
case, the
adapter replaces all trailing arguments from the original
trailing
argument position onward, by a new array of type arrayType, 
whose

elements comprise (in order) the replaced arguments.
___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: [9] RFR (S): 8036117: MethodHandles.catchException doesn't handle VarargsCollector right (8034120 failed)

[Peter Levart]

Hi,

Excuse my ignorant question: What is the purpose of @LambdaForm.Hidden 
annotation?


I suspect it has to do with hiding the call frames in stack traces that 
are part of LambdaForm invocation chain. In this case, method:


private static Object[] prepend(Object elem, Object[] array)

in MethodHandleImpl need not be annotated with this annotation, since 
it's call frame is not on stack when one of the target methods is 
executed. It's just a function used to calculate the argument of the 
call. In fact, if prepend() ever throws exception (NPE in case array is 
null?), It would be preferable that it's call frame is visible in the 
stack trace.


Am I right or am I just talking nonsense?

Regards, Peter


On 03/11/2014 12:12 AM, Vladimir Ivanov wrote:

John, Chris, thanks!

Best regards,
Vladimir Ivanov

On 3/11/14 3:08 AM, Christian Thalinger wrote:

Even better.

On Mar 10, 2014, at 3:21 PM, Vladimir Ivanov 
vladimir.x.iva...@oracle.com wrote:



Chris, thanks for the review.

John suggested an elegant way to fix the problem - use asFixedArity.

Updated fix:
http://cr.openjdk.java.net/~vlivanov/8036117/webrev.01/

Best regards,
Vladimir Ivanov

On 3/8/14 4:51 AM, Christian Thalinger wrote:

Seems good to me.  I’d like to have another name for this method:

+ private static Object invokeCustom(MethodHandle target, 
Object... args) throws Throwable {


On Mar 4, 2014, at 12:00 PM, Vladimir Ivanov 
vladimir.x.iva...@oracle.com wrote:



http://cr.openjdk.java.net/~vlivanov/8036117/webrev.00/
https://bugs.openjdk.java.net/browse/JDK-8036117
84 lines changed: 74 ins; 3 del; 7 mod

I have to revert a cleanup I did for 8027827.
MethodHandle.invokeWithArguments (and generic invocation) has 
unpleasant

peculiarity in behavior when used with VarargsCollector. So,
unfortunately, invokeWithArguments is not an option there.

Looking at the API (excerpts from javadoc [1] [2]), the following
condition doesn't hold in that case:
   trailing parameter type of the caller is a reference type 
identical

to or assignable to the trailing parameter type of the adapter.

Example:
   target.invokeWithArguments((Object[])args)
   =
   target.invoke((Object)o1,(Object)o2,(Object)o3)
   =/
   target.invokeExact((Object)o1, (Object)o2, (Object[])o3)

because Object !: Object[].

The fix is to skip unnecessary conversion when invoking a method 
handle

and just do a pairwise type conversion.

Testing: failing test case, nashorn w/ experimental features (octane)

Thanks!

Best regards,
Vladimir Ivanov

[1] MethodHandle.invokeWithArguments
Performs a variable arity invocation, ..., as if via an inexact 
invoke
from a call site which mentions only the type Object, and whose 
arity is

the length of the argument array.

[2] MethodHandle.asVarargsCollector
When called with plain, inexact invoke, if the caller type is the 
same

as the adapter, the adapter invokes the target as with invokeExact.
(This is the normal behavior for invoke when types match.)

Otherwise, if the caller and adapter arity are the same, and the
trailing parameter type of the caller is a reference type 
identical to

or assignable to the trailing parameter type of the adapter, the
arguments and return values are converted pairwise, as if by 
asType on a

fixed arity method handle.

Otherwise, the arities differ, or the adapter's trailing parameter 
type
is not assignable from the corresponding caller type. In this 
case, the

adapter replaces all trailing arguments from the original trailing
argument position onward, by a new array of type arrayType, whose
elements comprise (in order) the replaced arguments.
___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: [9] RFR (S): 8036117: MethodHandles.catchException doesn't handle VarargsCollector right (8034120 failed)

[Peter Levart]

On 03/11/2014 03:05 PM, Vladimir Ivanov wrote:

You raised totally valid question.

I marked MethodHandleImpl.prepend with @Hidden annotation because it
is internal implementation detail of JSR292.

You are right that normally a callee can't see it on stack. *But it's
possible to observe it when stack trace is queried from a separate
thread. *


Is this good or bad? It enables tools to see it (for example sampling
profilers, etc...).
For the purposes of sampling profilers (and other monitoring tools) 
@Hidden should be completely ignored.  These tools should use 
appropriate API to get the data.


I looked into the code and @Hidden has even less effect than I thought 
initially. It affects very limited set of cases - only users of 
JVM_FillInStackTrace (when filling exception's stack trace, 
Thread.dumpStack()). Calls to Thread.getStackTrace() from a separate 
thread omit stack trace filtering.


So all you're achieving by annotating prepend() method is that any 
exception stack trace, in case it is thrown inside the prepend() method, 
will hide where it was thrown from. In case all LambdaForm frames 
leading to the prepend() method were also hidden, the exception would 
appear to be thrown from the invocation of the MH...


Regards, Peter



Best regards,
Vladimir Ivanov


Regards, Peter



There's not much value in it in this particular case, but I decided to
reduce possible noise.

Best regards,
Vladimir Ivanov

On 3/11/14 3:35 PM, Peter Levart wrote:

Hi,

Excuse my ignorant question: What is the purpose of @LambdaForm.Hidden
annotation?

I suspect it has to do with hiding the call frames in stack traces 
that

are part of LambdaForm invocation chain. In this case, method:

 private static Object[] prepend(Object elem, Object[] array)

in MethodHandleImpl need not be annotated with this annotation, since
it's call frame is not on stack when one of the target methods is
executed. It's just a function used to calculate the argument of the
call. In fact, if prepend() ever throws exception (NPE in case 
array is

null?), It would be preferable that it's call frame is visible in the
stack trace.

Am I right or am I just talking nonsense?

Regards, Peter


On 03/11/2014 12:12 AM, Vladimir Ivanov wrote:

John, Chris, thanks!

Best regards,
Vladimir Ivanov

On 3/11/14 3:08 AM, Christian Thalinger wrote:

Even better.

On Mar 10, 2014, at 3:21 PM, Vladimir Ivanov
vladimir.x.iva...@oracle.com wrote:


Chris, thanks for the review.

John suggested an elegant way to fix the problem - use 
asFixedArity.


Updated fix:
http://cr.openjdk.java.net/~vlivanov/8036117/webrev.01/

Best regards,
Vladimir Ivanov

On 3/8/14 4:51 AM, Christian Thalinger wrote:

Seems good to me.  I’d like to have another name for this method:

+ private static Object invokeCustom(MethodHandle target,
Object... args) throws Throwable {

On Mar 4, 2014, at 12:00 PM, Vladimir Ivanov
vladimir.x.iva...@oracle.com wrote:


http://cr.openjdk.java.net/~vlivanov/8036117/webrev.00/
https://bugs.openjdk.java.net/browse/JDK-8036117
84 lines changed: 74 ins; 3 del; 7 mod

I have to revert a cleanup I did for 8027827.
MethodHandle.invokeWithArguments (and generic invocation) has
unpleasant
peculiarity in behavior when used with VarargsCollector. So,
unfortunately, invokeWithArguments is not an option there.

Looking at the API (excerpts from javadoc [1] [2]), the following
condition doesn't hold in that case:
   trailing parameter type of the caller is a reference type
identical
to or assignable to the trailing parameter type of the adapter.

Example:
   target.invokeWithArguments((Object[])args)
   =
   target.invoke((Object)o1,(Object)o2,(Object)o3)
   =/
   target.invokeExact((Object)o1, (Object)o2, (Object[])o3)

because Object !: Object[].

The fix is to skip unnecessary conversion when invoking a method
handle
and just do a pairwise type conversion.

Testing: failing test case, nashorn w/ experimental features
(octane)

Thanks!

Best regards,
Vladimir Ivanov

[1] MethodHandle.invokeWithArguments
Performs a variable arity invocation, ..., as if via an inexact
invoke
from a call site which mentions only the type Object, and whose
arity is
the length of the argument array.

[2] MethodHandle.asVarargsCollector
When called with plain, inexact invoke, if the caller type is 
the

same
as the adapter, the adapter invokes the target as with 
invokeExact.

(This is the normal behavior for invoke when types match.)

Otherwise, if the caller and adapter arity are the same, and the
trailing parameter type of the caller is a reference type
identical to
or assignable to the trailing parameter type of the adapter, the
arguments and return values are converted pairwise, as if by
asType on a
fixed arity method handle.

Otherwise, the arities differ, or the adapter's trailing 
parameter

type
is not assignable from the corresponding caller type. In this
case, the
adapter replaces all trailing arguments from the original 
trailing

argument position onward

MethodHandleInfo.getDeclaringClass()

[Peter Levart]

Hi,

The following code:

public class Test {

static class X {
public void x() { }
}

static class Y extends X {
}

public static void main(String[] args) throws Exception {
MethodHandles.Lookup lookup = MethodHandles.lookup();
MethodHandle mh = lookup.findVirtual(Y.class, x, 
MethodType.methodType(void.class, new Class[0]));

MethodHandleInfo mhi = lookup.revealDirect(mh);
System.out.println(mhi.getDeclaringClass().getSimpleName());
}
}


...prints Y. Is this correct? The javadoc for the method says:


/Returns the class in which the cracked method handle's underlying 
member was defined.//

//
//Returns://
//the declaring class of the underlying member/


The declaring class of the underlying member (the method x()) is 
clearly not class Y,  but class X, isn't it?


Regards, Peter

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: Signature of MethodHandleInfo.reflectAs is not specific enough

[Peter Levart]

On 11/11/2013 08:14 AM, Peter Levart wrote:
 The method could simply be:

 public T extends Member T reflect(Lookup lookup);

But if one needs to hint the compiler, explicit type parameters can be 
used as an escape hatch as always:

 Object o = info.Methodreflect(lookup);


Well, well, explicit type parameters for method invocation are not 
needed in above example:

 Object o = info.reflect(lookup); // compiles

One would only need them in situations like:

 info.Methodreflect(lookup).invoke(...);


Regards, Peter

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: Signature of MethodHandleInfo.reflectAs is not specific enough

[Peter Levart]

On 11/11/2013 02:24 AM, Ali Ebrahimi wrote:

This is another workaround:

public T extends MemberAnnotatedElement, R R reflectAs(Class? 
super T expected, Lookup lookup);


info.reflectAs(Member.class, lookup);//works
info.reflectAs(AnnotatedElement.class, lookup);//works
info.reflectAs(Member.class, lookup);//works
info.reflectAs(AnnotatedElement.class, lookup);//works


info.reflectAs(Object.class, lookup);doesn't work.
info.reflectAs(Other.class, lookup);doesn't work.

with this does not need to your javadoc and is more type safe. .


Hm... it doesn't look very compile-time type-safe:

String s = info.reflectAs(Method.class, lookup); // compiles !!!


IMO, I would rather remove the Class parameter altogether. It serves no 
purpose in the method implementation other than to perform a cast-check 
on the returned object. The method could simply be:


public T extends Member T reflect(Lookup lookup);

This would not solve the problem Remi put forward. I.e. this would not 
compile:


AnnotatedElement ae = info.reflect(lookup);

But with an explicit cast, It compiles:

AnnotatedElement ae = (AnnotatedElement) info.reflect(lookup);

And compared to what we would have with Class parameter and loosened 
compile-time type-safety as per Remi's suggestion, it is still shorter:


AnnotatedElement ae = info.reflectAs(AnnotatedElement.class, 
lookup); // this is longer!


A type-unsafe variant is possible too (I'm not advocating it):

public T T reflect(Lookup lookup);

Now that Generalized Target-Type Inference 
http://openjdk.java.net/projects/jdk8/features#101 is part of Java 8, 
using ClassT parameters just as hints to the compiler is not needed in 
many cases. But if one needs to hint the compiler, explicit type 
parameters can be used as an escape hatch as always:


Object o = info.Methodreflect(lookup);


Regards, Peter






On Mon, Nov 11, 2013 at 1:59 AM, Remi Forax fo...@univ-mlv.fr 
mailto:fo...@univ-mlv.fr wrote:


The is a stupid issue with the signature of
MethodHandleInfo.reflectAs,
j.l.r.Field, Method or Constructor implement two interfaces Member and
AnnotatedElement, with the current signature, the code
   info.reflectAs(Member.class, lookup)
works but the code
   info.reflectAs(AnnotatedElement.class, lookup)
doesn't work.

Because there is no way to do an 'or' between several bounds of
a type variable, I think that the signature of reflectAs should be
changed from :
   public T extends Member T reflectAs(ClassT expected, Lookup
lookup);
to
   public T T reflectAs(ClassT expected, Lookup lookup);

and the javadoc should be modified to explain that a Member or
AnnotatedElement are
valid bounds of T.

As a side effect, the signature of MethodHandles.reflectAs(ClassT,
MethodHandle)
should be updated accordingly.

There is a workaround, one can write:
   (AnnotatedElement)info.reflectAs(Member.class, lookup)
but it's at best weird.

cheers,
Rémi

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net mailto:mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev




___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: Classes on the stack trace

[Peter Levart]

On 07/31/2013 03:18 PM, Nick Williams wrote:
Okay, but I don't think @CallerSensitive is necessary to prevent 
MethodHandles from spoofing caller classes. I'll admit I don't know 
very much about MethodHandle, what it does, or how its invocation 
methods are different from Method.invoke(). In the code that I copied 
into this email from jvm.cpp last night, the third frame (frame #2) 
will always be returned unless it is one ignored by the security stack 
walk. @CallerSensitive doesn't come in to play here.


Nope. I think @CallerSensitive annotation is currently just a security 
feature, needed for implementation of MethodHandles.


Regards, Peter



Nick

On Jul 31, 2013, at 2:36 AM, Peter Levart wrote:


Hi Nick,

The @CallerSensitive annotation is an annotation that prevents some 
other infrastructure, namely the MethodHandles, to spoof caller 
classes.


Try this:

MethodHandles.Lookup lookup = 
MethodHandles.lookup().in(Object.class);
MethodHandle mh = lookup.findStatic(Class.class, forName, 
MethodType.methodType(Class.class, String.class));


...you won't be able to pretend that you are the j.l.Object that is 
calling method Class.forName(String)...


The annotation might have (or will have?) other effects like making 
sure some infrastructure-inserted frames are hidden-away just for 
@CallerSensitive methods which might be less optimal and not needed 
for normal methods.



Regards, Peter

On 07/31/2013 01:11 AM, Nick Williams wrote:

Quick question for those of you that know anything about @CallerSensitive...

After looking at the code and experimenting some, I've discovered that 
getCallerClass() doesn't actually keep going until it finds the first method 
without @CallerSensitive. It only returns the caller of the caller. So, for 
example:

Stack 1
@CallerSensitive Reflection.getCallerClass()
@CallerSensitive MyClass1.method1();
MyClass2.method2();

In this case, getCallerClass() returns MyClass2.class. BUT:

Stack 2
@CallerSensitive Reflection.getCallerClass()
@CallerSensitive MyClass1.method1();
@CallerSensitive MyClass2.method2();
MyClass3.method3();

In this case, getCallerClass() STILL returns MyClass2.class. Based on the plain-language 
meaning of @CallerSensitive, I would expect getCallerClass() to return MyClass3.class in 
the second case. But, indeed, the JavaDoc for Reflection.getCallerClass() says: 
Returns the class of the caller of the method calling this method. So, then, 
what's the point of @CallerSensitive? Looking at the code:

vframeStream vfst(thread);
// Cf. LibraryCallKit::inline_native_Reflection_getCallerClass
for (int n = 0; !vfst.at_end(); vfst.security_next(), n++) {
  Method* m = vfst.method();
  assert(m != NULL, sanity);
  switch (n) {
  case 0:
// This must only be called from Reflection.getCallerClass
if (m-intrinsic_id() != vmIntrinsics::_getCallerClass) {
  THROW_MSG_NULL(vmSymbols::java_lang_InternalError(), JVM_GetCallerClass 
must only be called from Reflection.getCallerClass);
}
// fall-through
  case 1:
// Frame 0 and 1 must be caller sensitive.
if (!m-caller_sensitive()) {
  THROW_MSG_NULL(vmSymbols::java_lang_InternalError(), 
err_msg(CallerSensitive annotation expected at frame %d, n));
}
break;
  default:
if (!m-is_ignored_by_security_stack_walk()) {
  // We have reached the desired frame; return the holder class.
  return (jclass) JNIHandles::make_local(env, 
m-method_holder()-java_mirror());
}
break;
  }
}

It seems to me that @CallerSensitive is completely pointless. This is ALWAYS 
going to return the first non-reflection frame after frame 1, regardless of 
@CallerSensitive. If @CallerSensitive were really supposed to have an actual 
purpose, it would seem to me that the last part should be this:

if (!m-is_ignored_by_security_stack_walk()  !m-caller_sensitive()) {
  // We have reached the desired frame; return the holder class.
  return (jclass) JNIHandles::make_local(env, 
m-method_holder()-java_mirror());
}

Am I completely missing the point here? I just don't see a reason for 
@CallerSensitive. The code could do the exact same thing it currently is 
without @CallerSensitive (except for enforcing that frame 1 is 
@CallerSensitive, which really isn't necessary if you aren't using it in 
further frames).

Thoughts?

Nick

On Jul 30, 2013, at 10:33 AM, Jochen Theodorou wrote:


Am 30.07.2013 16:16, schrieb Peter Levart:

On 07/30/2013 03:19 PM, Jochen Theodorou wrote:

Am 30.07.2013 14:17, schrieb Peter Levart:
[...]

So what would give Groovy or other language runtimes headaches when all
there was was a parameter-less getCallerClass() API? Aren't the
intermediate frames inserted by those runtimes controlled by the
runtimes? Couldn't the surface runtime-inserted methods capture the
caller and pass it down? I guess the problem

Re: Classes on the stack trace

[Peter Levart]

Hi Nick,

The @CallerSensitive annotation is an annotation that prevents some 
other infrastructure, namely the MethodHandles, to spoof caller classes.


Try this:

MethodHandles.Lookup lookup = 
MethodHandles.lookup().in(Object.class);
MethodHandle mh = lookup.findStatic(Class.class, forName, 
MethodType.methodType(Class.class, String.class));


...you won't be able to pretend that you are the j.l.Object that is 
calling method Class.forName(String)...


The annotation might have (or will have?) other effects like making sure 
some infrastructure-inserted frames are hidden-away just for 
@CallerSensitive methods which might be less optimal and not needed for 
normal methods.



Regards, Peter

On 07/31/2013 01:11 AM, Nick Williams wrote:

Quick question for those of you that know anything about @CallerSensitive...

After looking at the code and experimenting some, I've discovered that 
getCallerClass() doesn't actually keep going until it finds the first method 
without @CallerSensitive. It only returns the caller of the caller. So, for 
example:

Stack 1
@CallerSensitive Reflection.getCallerClass()
@CallerSensitive MyClass1.method1();
MyClass2.method2();

In this case, getCallerClass() returns MyClass2.class. BUT:

Stack 2
@CallerSensitive Reflection.getCallerClass()
@CallerSensitive MyClass1.method1();
@CallerSensitive MyClass2.method2();
MyClass3.method3();

In this case, getCallerClass() STILL returns MyClass2.class. Based on the plain-language 
meaning of @CallerSensitive, I would expect getCallerClass() to return MyClass3.class in 
the second case. But, indeed, the JavaDoc for Reflection.getCallerClass() says: 
Returns the class of the caller of the method calling this method. So, then, 
what's the point of @CallerSensitive? Looking at the code:

vframeStream vfst(thread);
// Cf. LibraryCallKit::inline_native_Reflection_getCallerClass
for (int n = 0; !vfst.at_end(); vfst.security_next(), n++) {
  Method* m = vfst.method();
  assert(m != NULL, sanity);
  switch (n) {
  case 0:
// This must only be called from Reflection.getCallerClass
if (m-intrinsic_id() != vmIntrinsics::_getCallerClass) {
  THROW_MSG_NULL(vmSymbols::java_lang_InternalError(), JVM_GetCallerClass 
must only be called from Reflection.getCallerClass);
}
// fall-through
  case 1:
// Frame 0 and 1 must be caller sensitive.
if (!m-caller_sensitive()) {
  THROW_MSG_NULL(vmSymbols::java_lang_InternalError(), 
err_msg(CallerSensitive annotation expected at frame %d, n));
}
break;
  default:
if (!m-is_ignored_by_security_stack_walk()) {
  // We have reached the desired frame; return the holder class.
  return (jclass) JNIHandles::make_local(env, 
m-method_holder()-java_mirror());
}
break;
  }
}

It seems to me that @CallerSensitive is completely pointless. This is ALWAYS 
going to return the first non-reflection frame after frame 1, regardless of 
@CallerSensitive. If @CallerSensitive were really supposed to have an actual 
purpose, it would seem to me that the last part should be this:

if (!m-is_ignored_by_security_stack_walk()  !m-caller_sensitive()) {
  // We have reached the desired frame; return the holder class.
  return (jclass) JNIHandles::make_local(env, 
m-method_holder()-java_mirror());
}

Am I completely missing the point here? I just don't see a reason for 
@CallerSensitive. The code could do the exact same thing it currently is 
without @CallerSensitive (except for enforcing that frame 1 is 
@CallerSensitive, which really isn't necessary if you aren't using it in 
further frames).

Thoughts?

Nick

On Jul 30, 2013, at 10:33 AM, Jochen Theodorou wrote:


Am 30.07.2013 16:16, schrieb Peter Levart:

On 07/30/2013 03:19 PM, Jochen Theodorou wrote:

Am 30.07.2013 14:17, schrieb Peter Levart:
[...]

So what would give Groovy or other language runtimes headaches when all
there was was a parameter-less getCallerClass() API? Aren't the
intermediate frames inserted by those runtimes controlled by the
runtimes? Couldn't the surface runtime-inserted methods capture the
caller and pass it down? I guess the problem is supporting calling the
caller-sensitive methods like Class.forName(String) and such which don't
have the overloaded variant taking caller Class or ClassLoader as an
argument...

Speaking for Groovy...
those intermediate frames are runtime controlled, yes, but passing down
the caller class is exactly the problem. Imagine I would suggest that
each and every method definition in Java automatically gets an
additional parameter for the caller class, just to have access to it
inside the method. You would not accept that for Java, would you? And so
we cannot accept that for Groovy if we want to keep integration with
Java...

Are you talking about internal Groovy implementation (the
runtime-inserted methods) or the publicly

Re: Classes on the stack trace (was: getElementClass/StackTraceElement, was: @CallerSensitive public API, was: sun.reflect.Reflection.getCallerClass)

[Peter Levart]

On 07/27/2013 09:01 PM, Nick Williams wrote:

All,

In the last two months, there have been a number of discussions surrounding 
stack traces, Classes on the stack trace, and caller classes [1], [2], [3]. 
These are all related discussions and the solution to them is equally related, 
so I wanted to consolidate it all into this one discussion where I hope we can 
finalize on a solution and get it implemented for Java 8.

In a nut shell, here are the underlying needs that I have seen expressed 
through many, many messages:

- Some code needs to get the Class of the caller of the current method, 
skipping any reflection methods.
- Some code needs to get the Class of the caller /n/ stack frames before the 
current method, skipping any reflection methods.
- Some code needs to get the current stack trace, populated with Classes, 
Executables, file names, line numbers, and native flags instead of the String 
class names and String method names in StackTraceElement. This /should/ include 
any reflection methods, just like StackTraceElement[]s.
- Some code needs to get the stack trace from when a Throwable was created, 
populated with Classes, Executables, file names, line numbers, and native flags 
instead of the String class names and String method names in StackTraceElement. 
This /should/ include any reflection methods, just like StackTraceElement[]s.
- There needs to be a reflection way to achieve all of this since some 
libraries (e.g., Log4j) need to be compiled against Java 6 but run on 7 and 8 
(and thus can't use @CallerSensitive).

I believe the solutions to these needs are all related. Importantly, I think it is 
very important that action be taken in Java 8 due to the changes made to 
sun.reflect.Reflection#getCallerClass(...). While we all understand that relying 
on private sun.* APIs is not safe, the fact is that many people have relied on 
sun.reflect.Reflection#getCallerClass(...) due to the fact that there is simply no 
other way to do this in the standard API. This includes Log4j 2, Logback, SLF4j, 
and Groovy, some features of which will stop working correctly in Java 7 = u25.


Hi,

The needs described above may seem related, but from what I see in this 
commit:


http://hg.openjdk.java.net/jdk8/tl/jdk/rev/da6addef956e

my observations are as following (please comment if I missed or 
misunderstood anything):


sun.reflect.Reflection.getCallerClass(int) is/was used internally in JDK 
more or less for different purposes than outside the JDK. Inside it was 
used basically for implementing security-sensitive checks like 
optimizations for public methods which can avoid calling SecurityManager 
API wen called from withing JDK classes. It was used for 
security-unrelated purposes too, like for example Class.forName(String) 
or ResourceBundle.getBundle(String). All internal JDK uses do share one 
common thing though: it is very important that the right direct caller 
of a caller-sensitive method is established, since any failure to do so 
can have devastating effect on security or correctness.


The API taking an int to count the frames between the top of the 
call-stack to the indirect caller was convenient, but too fragile to 
support such important use cases. Every time some code was refactored, 
there was danger that some call-frame was inadvertently inserted or 
removed. So I think it was decided to cripple the API to only support 
obtaining the immediate caller of the method making call to the 
Reflection.getCallerClass() and all uses modified accordingly to make 
the internal JDK code more robust to refactorings.


And there's also MethodHandles which are early-bound. Meaning that the 
caller is established and bound when the MethodHandle instance is 
looked-up. The lookupClass which is associated with the Lookup object 
and used in permission checks when obtaining MHs is also used as the 
bound caller class when the MH is invoked. Now there's a method:


java.lang.invoke.MethodHandles.Lookup {
public java.lang.invoke.MethodHandles.Lookup in(java.lang.Class? 
requestedLookupClass)


that returns a Lookup object which reports a different lookupClass and 
has capabilities to lookup MHs which are combined from capabilities of 
the original Lookup object and new lookupClass (tipicaly less, never 
more). Most of such Lookup objects are prevented from looking-up MHs for 
caller-sensitive methods, since they could be used to pose as a caller 
that is not the one having obtained the MH and therefore gain access to 
restricted resource, for example:


MethodHandle mh = MethodHandles.lookup().in(Object.class).lookupXXX()

...such mh could be used to pose as being called from withing Object if 
allowed to be obtained for caller-sensitive methods. So here comes 
@CallerSensitive annotation to mark such methods and prevent such 
lookups (before that - in JDK7, all internal caller-sensitive methods 
were hand-maintained in a list).


So this is, what I think, the background and rationale for 

Re: Classes on the stack trace (was: getElementClass/StackTraceElement, was: @CallerSensitive public API, was: sun.reflect.Reflection.getCallerClass)

[Peter Levart]

I think there are various techniques to use new API when running on JDK8 
and still use old API when running on JDK6 or 7. One way is to have 2 
classes implementing the same interface or extending basic class where 
one is compiled with JDK6/7 and the other with JDK8. During runtime the 
correct implementation is choosen, but code only depends on the 
interface/basic class...


There are other tricks.

But I have an idea. What if Reflection.getCallerClass(int) is restored 
in JDK7 and JDK8 without a @CallerSensitive annotation on it. In order 
to prevent inadvertent internal JDK usage, the method could throw 
exception when called from any internal JDK class...


Regards, Peter


On 07/30/2013 02:32 PM, Nick Williams wrote:
For caller-sensitive methods, the approach taken with new 
Reflection.getCallerClass() is the right one, I think. There's no need 
to support a fragile API when caller-sensitivity is concerned, so the 
lack of int parameter, combined with annotation for marking such 
methods is correct approach, I think.


Not when the code running on Java 8 was compiled on Java 6 or 7 and 
thus can't be annotated @CallerSensitive. In these cases, use of any 
new public API must use reflection (sudo code: If Java 8, use new 
public caller-sensitive API), so it needs code that it can pass a 
number into.


Nick

On Jul 30, 2013, at 7:17 AM, Peter Levart wrote:



On 07/27/2013 09:01 PM, Nick Williams wrote:

All,

In the last two months, there have been a number of discussions surrounding 
stack traces, Classes on the stack trace, and caller classes [1], [2], [3]. 
These are all related discussions and the solution to them is equally related, 
so I wanted to consolidate it all into this one discussion where I hope we can 
finalize on a solution and get it implemented for Java 8.

In a nut shell, here are the underlying needs that I have seen expressed 
through many, many messages:

- Some code needs to get the Class of the caller of the current method, 
skipping any reflection methods.
- Some code needs to get the Class of the caller /n/ stack frames before the 
current method, skipping any reflection methods.
- Some code needs to get the current stack trace, populated with Classes, 
Executables, file names, line numbers, and native flags instead of the String 
class names and String method names in StackTraceElement. This /should/ include 
any reflection methods, just like StackTraceElement[]s.
- Some code needs to get the stack trace from when a Throwable was created, 
populated with Classes, Executables, file names, line numbers, and native flags 
instead of the String class names and String method names in StackTraceElement. 
This /should/ include any reflection methods, just like StackTraceElement[]s.
- There needs to be a reflection way to achieve all of this since some 
libraries (e.g., Log4j) need to be compiled against Java 6 but run on 7 and 8 
(and thus can't use @CallerSensitive).

I believe the solutions to these needs are all related. Importantly, I think it is 
very important that action be taken in Java 8 due to the changes made to 
sun.reflect.Reflection#getCallerClass(...). While we all understand that relying 
on private sun.* APIs is not safe, the fact is that many people have relied on 
sun.reflect.Reflection#getCallerClass(...) due to the fact that there is simply no 
other way to do this in the standard API. This includes Log4j 2, Logback, SLF4j, 
and Groovy, some features of which will stop working correctly in Java 7 = u25.


Hi,

The needs described above may seem related, but from what I see in 
this commit:


http://hg.openjdk.java.net/jdk8/tl/jdk/rev/da6addef956e

my observations are as following (please comment if I missed or 
misunderstood anything):


sun.reflect.Reflection.getCallerClass(int) is/was used internally in 
JDK more or less for different purposes than outside the JDK. Inside 
it was used basically for implementing security-sensitive checks like 
optimizations for public methods which can avoid calling 
SecurityManager API wen called from withing JDK classes. It was used 
for security-unrelated purposes too, like for example 
Class.forName(String) or ResourceBundle.getBundle(String). All 
internal JDK uses do share one common thing though: it is very 
important that the right direct caller of a caller-sensitive method 
is established, since any failure to do so can have devastating 
effect on security or correctness.


The API taking an int to count the frames between the top of the 
call-stack to the indirect caller was convenient, but too fragile to 
support such important use cases. Every time some code was 
refactored, there was danger that some call-frame was inadvertently 
inserted or removed. So I think it was decided to cripple the API 
to only support obtaining the immediate caller of the method making 
call to the Reflection.getCallerClass() and all uses modified 
accordingly to make the internal JDK code more robust to refactorings.


And there's

Re: Classes on the stack trace (was: getElementClass/StackTraceElement, was: @CallerSensitive public API, was: sun.reflect.Reflection.getCallerClass)

[Peter Levart]

On 07/30/2013 03:19 PM, Jochen Theodorou wrote:

Am 30.07.2013 14:17, schrieb Peter Levart:
[...]

So what would give Groovy or other language runtimes headaches when all
there was was a parameter-less getCallerClass() API? Aren't the
intermediate frames inserted by those runtimes controlled by the
runtimes? Couldn't the surface runtime-inserted methods capture the
caller and pass it down? I guess the problem is supporting calling the
caller-sensitive methods like Class.forName(String) and such which don't
have the overloaded variant taking caller Class or ClassLoader as an
argument...

Speaking for Groovy...
those intermediate frames are runtime controlled, yes, but passing down
the caller class is exactly the problem. Imagine I would suggest that
each and every method definition in Java automatically gets an
additional parameter for the caller class, just to have access to it
inside the method. You would not accept that for Java, would you? And so
we cannot accept that for Groovy if we want to keep integration with
Java...


Are you talking about internal Groovy implementation (the 
runtime-inserted methods) or the publicly visible API? One solution for 
internal implementation of Groovy could be (speaking by heart since I 
don't know the internals of Groovy) for the surface public API method 
which doesn't have to have the special caller parameter, to capture the 
caller with getCallerClass() parameterless API (possibly enclosed with a 
quick check confirming that it might actually be needed) and bind it to 
a ThreadLocal, then use this ThreadLocal down at the end...



and the good integration with Java is one of the key points of
Groovy. Even if we make something like that @CallerSensitive and add the
parameter only in those cases, we break being able to override methods.


I guess I don't know every Groovy need to obtain the caller class. I 
thought the problem was to support calling caller-sensitive methods in 
Java API (like Class.forName(String)) from within Groovy code, where 
there are runtime-inserted frames between the call-site and the target 
method. Are there any other needs?



Plus, before Groovy3 is not done we have to support several call paths.
And the oldest one, which is still a fallback, does not support
transporting the caller class through the runtime layers at all.
Changing here is a breaking change.


Could you describe those call-paths? Examples of Groovy code and to what 
it gets translated (equivalent Java code at call site) with a brief 
description of what each intermediate layer (between the call-site and 
the target method) does and at which point the caller class is extracted...


Regards, Peter


John Rose suggested to capture the caller in the surface method and
bind it with a MethodHandle and then pass such MH down the runtime API
and finally call that method via MH.

Only that you then need a java7+ only version, plus the already
mentioned problem, that not all of the three general call paths support
that or can be changed in a way to enable that without breaking user code.


bye Jochen



___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: Classes on the stack trace (was: getElementClass/StackTraceElement, was: @CallerSensitive public API, was: sun.reflect.Reflection.getCallerClass)

[Peter Levart]

On 07/30/2013 06:46 PM, MacGregor, Duncan (GE Energy Management) wrote:

I can understand the worry about exposing @CallerSensitive methods, at
least for the case of lookup.in() as I'm sure it could be used to open an
exciting can of worms, but the current limitation doesn't feel quite
right. I wonder whether solution could go something like:

  1. I have a lookup from an invokeDynamic bootstrap. This should be
full-powered with its lookupClass being the class containing the
invokeDynamic.
  2. Getting a new Lookup object using the lookup.in() method should return
something that can lookup @CallerSensitive methods that are visible to it,
but it's 'caller' remains that of the original lookup class.


I view lookup.in(requestedLookupClass) as a means to change the lookup 
class *and* the caller class at the same time. I think the new Lookup 
object obtained this way is never more capable of looking-up members 
than the original Lookup object but can be less capable, so if you 
already have the Lookup object avaliable from the invokeDynamic 
bootstrap, then this is the right object to use.


The case I'm after is when there's no invokeDynamic involved and the 
caller class must be obtained in the caller-sensitive way (by some 
Reflection.getCallerClass() method or such). It would be useful if under 
some constraints (perhaps the visibility of called class from the 
calling class) the called method which would be marked as 
caller-sensitive could obtain a Lookup object for the calling class 
which would allow looking up other @CallerSensitive methods, like:


@CallerSensitive
public void myCallerSensitiveMethod() {
Lookup callerLookup = 
MethodHandles.lookup().in(Reflection.getCallerClass());

MethodHandle targetCallerSensitiveMH = callerLookup.lookupXXX();
targetCallerSensitiveMH.invoke(...);
}

Regards, Peter


Regards, Duncan.


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: method handle cracking API

[Peter Levart]

Hi John,

I peeked at the implementation of HMI.asMember method:

   111 +// Helper for default MethodHandleInfo.asMember.
   112 +static Member reflectMember(MethodHandleInfo self) 
throws ReflectiveOperationException {
   113 +int mods = self.getModifiers();
   114 +if (mods == -1)  throw notDirectMethodHandle();
   115 +byte refKind = (byte) self.getReferenceKind();
   116 +Class? defc = self.getDeclaringClass();
   117 +boolean isPublic = Modifier.isPublic(mods);
   118 +if (MethodHandleNatives.refKindIsMethod(refKind)) {
   119 +if (isPublic)
   120 +return defc.getMethod(self.getName(), 
self.getMethodType().parameterArray());
   121 +else
   122 +return 
defc.getDeclaredMethod(self.getName(), 
self.getMethodType().parameterArray());
   123 +} else if 
(MethodHandleNatives.refKindIsConstructor(refKind)) {
   124 +if (isPublic)
   125 +return 
defc.getConstructor(self.getMethodType().parameterArray());
   126 +else
   127 +return 
defc.getDeclaredConstructor(self.getMethodType().parameterArray());
   128 +} else if 
(MethodHandleNatives.refKindIsField(refKind)) {
   129 +if (isPublic)
   130 +return defc.getField(self.getName());
   131 +else
   132 +return defc.getDeclaredField(self.getName());
   133 +} else {
   134 +throw new 
IllegalArgumentException(referenceKind=+refKind);
   135 +}
   136 +}
...
   408 +public default T extends Member T asMember(ClassT 
expected) {
   409 +Member mem = AccessController.doPrivileged(new 
PrivilegedActionMember() {
   410 +public Member run() {
   411 +try {
   412 +return 
InfoFromMemberName.reflectMember(MethodHandleInfo.this);
   413 +} catch (ReflectiveOperationException ex) {
   414 +throw newInternalError(ex);
   415 +}
   416 +}
   417 +});
   418 +return expected.cast(mem);
   419  }


Are special cases for public members just because the reflection API 
requires different privileges for public vs. declared member access? But 
InfoFromMemberName.reflectMember() is package-private and is called from 
within a PrivilegedAction in MHI.asMember() which is a system class. If 
MHI objects are not considered security sensitive tokens like MH 
objects, then perhaps MHI.asMember should not wrap the call to 
InfoFromMemberName.reflectMember() in a PrivilegedAction...


Regards, Peter

On 05/15/2013 07:58 AM, John Rose wrote:
 Here is an update on the method handle cracking API.

 Though we will probably make some small adjustments, it appears to be 
 converging.

 http://cr.openjdk.java.net/~jrose/8008688/specdiff.02/
 http://cr.openjdk.java.net/~jrose/8008688/javadoc.02/

 Key points:

 - interface MethodHandleInfo (MHI) presents a cracked symbolic reference 
 for the member underlying a direct method handle (DMH)
 - cracking of method handles which are not DMHs is left for the future
 - you need the original Lookup object associated with DMH creation in order 
 to crack it; this prevents side-channels

 - there are access methods for the four components (ref kind, class, name, 
 type) of the CONSTANT_MethodHandle (or equivalent information) that was used 
 to create the DMH
 - there is a bridge (asMember) from MHI to the Core Reflection API (can be 
 built by user but very complex)
 - there is a fast path to the modifier bits of the underlying member (for 
 checking public, varargs, etc.)
 - there are two static utility methods on MHI, toString and 
 getReferenceKindString

 - there is also an unchecked version of the cracker 
 (revealDirectAsAccessibleObject), which always makes a security manager call 
 and bridges directly to the Core Reflection API

 - Modifier.isVarArgs and Modifier.VARARGS are made public, because there was 
 no standard test, and variable arity is a key MethodHandle property

 The implementation patch is refreshed:

 http://hg.openjdk.java.net/mlvm/mlvm/jdk/file/tip/meth-info-8008688.patch

 Changes since last E-mail:

 - added revealDirectAsAccessibleObject
 - revealDirectAsAccessibleObject performs the same strong SM check as 
 setAccessible
 - when either revealDirect function is called on a non-DMH, you get 
 NoSuchMethodException
 - refined the types and documentation of exceptions and throws
 - removed static methods from MHI, except getReferenceKindString and toString
 - moved isVarArgs to java.lang.reflect.Modifier, where it belongs

 Possible loose ends:

 - minor name changes
 - MHI.toMember needs a more precise spec, and the default 

Re: some thoughts on ClassValue

[Peter Levart]

On 04/27/2013 11:54 PM, John Rose wrote:
On Apr 27, 2013, at 2:34 PM, Jochen Theodorou blackd...@gmx.org 
mailto:blackd...@gmx.org wrote:



Am 27.04.2013 23:26, schrieb John Rose:
...
Each instance of the Groovy runtime will use a distinct ClassValue 
instance.


If the ClassValue instance goes dead, then all the values (for each
class) bound using that instance should go dead also.

If they don't it's a bug.


well... I assume that if I have a ClassValue for Object that is an
isntance of Foo, that this instance will stay as long as I don't delete
it myself. That is how ClassValue works, or not? And that is exactly my
problem too.


As a simple example:  Make a ClassValue object, bind a 1Mb array to 
Object.class using it, and then throw it all away.  (Except 
Object.class, naturally.)  Even if you do this in a loop, you should 
not get a storage leak.


(A class value bound to a class is live only if both the Class and the 
ClassValue instance are both live.)


Often people bind ClassValue instances to static final variables. 
 If the class containing the static variable goes away, then the 
ClassValue goes away, and anything bound (to whatever class, including 
Object) should go away.  That's nice, at least in theory.


Things get tricky (this probably happens in practice) when there is a 
loop:


  class MyRuntimeInMyClassLoader { static final MyClassValue MYCV = 
... } = instance of MYCV

  instance of MYCV x Object.class (say) = instance of MyMetaStuff
  instance of MyMetaStuff = MyRuntimeInMyClassLoader.class

Although I am not one of the 12 people on the planet who fully 
understand class loader interactions with GC, I suspect this loop 
might be sticky, and might have to be broken by (say) a 
SoftReference.  (You can use a SoftReference if you are willing to 
have the GC break references not recently used, which you can 
reconstruct if needed.)


--- John


Hello John, Jochen, sorry to interrupt,

So to break a loop, you propose the following structure if I understand 
correctly?


class MyRuntimeInMyClassLoader {
static final ClassValueSoftReferenceMyMetaStuff MYCV = new 
ClassValue() {...};



Wouldn't then MyRuntimeInMyClassLoader class be softly reachable (via 
softly reachable MyMetaStuff) and might not go away so quickly until 
there is memory pressure and consequently the MYCV ClassValue too?



Regards, Peter




___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: some thoughts on ClassValue

[Peter Levart]

On 04/28/2013 12:31 AM, Peter Levart wrote:


On 04/27/2013 11:54 PM, John Rose wrote:
On Apr 27, 2013, at 2:34 PM, Jochen Theodorou blackd...@gmx.org 
mailto:blackd...@gmx.org wrote:



Am 27.04.2013 23:26, schrieb John Rose:
...
Each instance of the Groovy runtime will use a distinct ClassValue 
instance.


If the ClassValue instance goes dead, then all the values (for each
class) bound using that instance should go dead also.

If they don't it's a bug.


well... I assume that if I have a ClassValue for Object that is an
isntance of Foo, that this instance will stay as long as I don't delete
it myself. That is how ClassValue works, or not? And that is exactly my
problem too.


As a simple example:  Make a ClassValue object, bind a 1Mb array to 
Object.class using it, and then throw it all away.  (Except 
Object.class, naturally.)  Even if you do this in a loop, you should 
not get a storage leak.


(A class value bound to a class is live only if both the Class and 
the ClassValue instance are both live.)


Often people bind ClassValue instances to static final variables. 
 If the class containing the static variable goes away, then the 
ClassValue goes away, and anything bound (to whatever class, 
including Object) should go away.  That's nice, at least in theory.


Things get tricky (this probably happens in practice) when there is a 
loop:


  class MyRuntimeInMyClassLoader { static final MyClassValue MYCV = 
... } = instance of MYCV

  instance of MYCV x Object.class (say) = instance of MyMetaStuff
  instance of MyMetaStuff = MyRuntimeInMyClassLoader.class

Although I am not one of the 12 people on the planet who fully 
understand class loader interactions with GC, I suspect this loop 
might be sticky, and might have to be broken by (say) a 
SoftReference.  (You can use a SoftReference if you are willing to 
have the GC break references not recently used, which you can 
reconstruct if needed.)


--- John


Hello John, Jochen, sorry to interrupt,

So to break a loop, you propose the following structure if I 
understand correctly?


class MyRuntimeInMyClassLoader {
static final ClassValueSoftReferenceMyMetaStuff MYCV = new 
ClassValue() {...};



Wouldn't then MyRuntimeInMyClassLoader class be softly reachable (via 
softly reachable MyMetaStuff) and might not go away so quickly until 
there is memory pressure and consequently the MYCV ClassValue too?



Regards, Peter


Perhaps the following could do:

class MyRuntimeInMyClassLoader {
static final ClassValueWeakReferenceMyMetaStuff MYCV = new 
ClassValue() {...};

// strongly reference all MyMetaStuff instances so they don't go away
// until MyRuntimeInMyClassLoader goes away
static final VectorMyMetaStuff metas = ...;

(idea taken from ClassLoader which retains it's Class objects in similar 
way)



Regards, Peter






___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev




___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: some thoughts on ClassValue

[Peter Levart]

On 04/26/2013 09:59 AM, Jochen Theodorou wrote:
 Hi all,

 basically ClassValue is there so that my runtime/framework or whatever
 can store information and bind it to a class. The information is then
 available as long as the class exists. In Groovy3 I am going to use this
 for meta classes.

 Some time ago a user came up with a special need, he wanted to be able
 to replace the groovy runtime at runtime but could not do so because of
 classes being referenced that keep classloaders alive, which then lock
 the jar on the file system on windows.

 This did let me to the thought, that as soon as I start storing meta
 classes using ClassValue, my Groovy runtime will never unload again as
 soon as I store for the example a meta class on ArrayList (which is
 likely to happen) or Object, or any other java core class for that
 matter. This makes me think that I should not directly use ClassValue to
 enable unloading... but I also do not want unloading to happen when I
 still need the value.

 Does anyone have any idea on now to solve this problem?

 bye blackdrag


Hi Jochen,

It seems you have similar issues that for example j.l.r.Proxy has. It 
spins a class implementing given interfaces, defines it with a given 
ClassLoader and then caches it for future-use. The situation with 
j.l.r.Proxy is simpler than your's, I assume. Could you describe in some 
more detail, what are meta classes in groovy runtime in terms of how 
they are generated (lazily at runtime or ahead of runtime?), what 
dependencies they can have and who is dependent on them. I assume they 
are generated at runtime, since if they were accessible as class files 
on the classpath of some loader, you would not have to design your own 
caching/loading/defining layer (Class.forName() would do). That's still 
an option if you create your own ClassLoader subclass that would use a 
naming convention (say groovymeta.some.pkg.ClassName would be the name 
of meta class belonging to some.pkg.ClassName main class) and use that 
ClassLoader as the class loader for groovy apps. But that's hard to 
enforce in various environments... So dynamic generation then.

You typically would want to define a class with the ClassLoader that 
also sees all the dependencies of the class. But there could be 
several ClassLoaders in a runtime that are suitable in a given 
situation. You would choose the right one depending on the lifetime of 
the generated class and the possible users of that class. And you would 
want the Class object of the meta class to be cached via a 
WeakReferenceClass, so that when all the users of the class are gone 
the class itself becomes one step to eligible to GC. It will not be 
GC-ed until the ClassLoader that defined it becomes eligible to GC 
(since the ClassLoader maintains explicit hard-references to all the 
Class objects for classes that it has defined). And each class maintains 
an implicit reference to the ClassLoader that defined it. So when all 
the users of all classes defined by a single ClassLoader are gone and 
the ClassLoader is not reachable any more, the whole graph of 
ClassLoader with all it's Class objects is GC-ed. It's that simple.

The question is how to most effectively design the structure so that you 
can quickly obtain the cached Class object and use the class represented 
by it. One part of the lookup key seems to be the main class that 
the meta class belongs to. So ClassValue is one option. But I can 
imagine you would want to support groovy runtime in environments such as 
app servers where the groovy runtime is deployed at the system level and 
the contents of a metaclass belonging to some system class (say 
ArrayList) is supplied by the deployed application and there could be 
several deployed applications, each contributing it's own different 
meta class for ArrayList. So the main class is not the only key for 
lookup. I would imagine the ClassLoader of the deployed application is 
the other part of the key and you would want to define a separate meta 
class with each application ClassLoader for the same main class. So the 
most general approach would be the structure similar to the following:


MapWeakReferenceClassLoader, ClassValueWeakReferenceClass
   ^  ^
   |  meta class
   the defining class loader of the meta class

which is equivalent to:

MapWeakReferenceClassLoader, MapWeakReferenceClass, 
WeakReferenceClass
   ^ ^ ^
   | | meta class
   |   main class
   the defining class loader of the meta class


  ... with the hashCode/equals of the WeakReferences suitably overriden 
of course.


Regards, Peter

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev

Re: another question about megamorphic call sites in combination with MethodHandles

[Peter Levart]

On 03/22/2013 11:50 AM, Remi Forax wrote:

On 03/22/2013 11:12 AM, Jochen Theodorou wrote:

Am 22.03.2013 10:35, schrieb Remi Forax:

On 03/22/2013 10:24 AM, Jochen Theodorou wrote:

Am 22.03.2013 10:11, schrieb Remi Forax:
[...]

I don't think it's a good idea to expose directly method handles to users,
it's better to encapsulate it into a Groovy object corresponding to a
function or a closure so you can add a bunch of invoke overloads.

what invoke overloads are you thinking of here?

traditionnal ones,
invoke(Object)
invoke(Object,Object)
...
invoke(Object...)

Well, I would probably use different name, to avoid the problem of
having a call with a single argument being an Object[] but then done
with invoke(Object...), which would use the array elements as arguments
instead of the array itself.

the usual trick to fall into that trap is to have two forms:
invoke(Object[]) and invoke(Object, Object...)


We have this problem in Closure#call
already, if done from Java... Which is why the method will go. Our
groovy.lang.Closure will become only a thin wrapper ideally.

Anyway... so you suggest having a general class we can use for
invocations, then make some method to request the invoker, which will
produce a subclass, with an implementation that can call the target?

In my opinion, your closure object should be a wrapper around a
MethodHandle.
You should keep the call method (and even add several overloads) as a
generic call mechanism with no inline optimisation. For the overload
like call(Object), the call will be like a function call,
so not something too bad.

And if a user want a call to a specific closure to be inlined, the user
should request a specific SAM implemented using a similar trick that the
Java lambda will do (maybe?).
The idea is to generate and reuse the same class for all instances of
the same SAM interface with that the call of a method of the interface
will be usually mono-morphic and inside the method of the generated
class an invokedynamic will call the method handle that is inside the
SAM proxy.
Because it's an invokedynamic, it can cache several method handle in
order to inline them with a guardWithTest that test method handle
instances instead of checking object class as usual.


Hi Remi,

So how is this going to work? I mean the dispatch. If one SAM class is 
going to dispatch to several MethodHandle instances, how would that be 
more optimal then calling an interface method on several implementations 
(classes implementing the method), each calling it's target method 
directly or using a constant MethodHandle?


Regards, Peter


It's basically a proxy generated once by SAM interface with an
invokedynamic inside.


bye blackdrag


cheers,
Rémi

___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev


___
mlvm-dev mailing list
mlvm-dev@openjdk.java.net
http://mail.openjdk.java.net/mailman/listinfo/mlvm-dev