Re: Reflection: how does one access a protected member in a superclass reflectively?

[John Rose]

On Jan 16, 2018, at 1:08 PM, Jochen Theodorou  wrote:
> 
> which is a big deal of a problem if the class, that is supposed to be exposed 
> to the framework is not under the control of the framework. And there is no 
> good solution (rewriting the bytecode for example is no good solution)

Yep.  We didn't solve all those problems in 9 since they are
very complicated trade-offs, and the "new kid on the block" of
enforced encapsulation is taking ground previously occupied
by the "neighborhood gang" of legitimate frameworks.  So we
need a way to give those frameworks what they need, in a way
that still make encapsulation preservable and checkable.

(My mental model for this tradeoff is, how can I AOT-compile as
much as possible, while still leaving open some legitimate hooks
for frameworks to operate?  I.e., enabling AOT and framework
intercession at the same time is what winning looks like.)

The privateLookup API will allow a framework to get full-power
"guest" status in an arbitrary uncooperative class.  At least,
that's the advertisement.  Will that help, or is there some bad
fine-print to that API?

Re: Reflection: how does one access a protected member in a superclass reflectively?

[John Rose]

On Jan 16, 2018, at 10:37 AM, Rony G. Flatscher  wrote:
> 
> Well that is probably the core of the problem: who needs to be the subclass, 
> the reflector or the
> object to be reflected upon (whose inheritance tree is being walked up 
> towards the root class, and
> if public or protected members are found reflectively accessed)?

Quick comment:

With the core reflection API, the reflector's permissions
are derived from the reflector, not the class being investigated.

This works fine when a class reflects itself but is not so good for frameworks.

The reflection API in java.lang.invoke, though more limited, does
support rights delegation through the Lookup object.  This means
that your "patient" class doesn't need to do the reflection work;
it can merely create a private lookup in itself and hand it to the
framework, which can do the reflection on behalf of the patient.

You can even mix the old and new mechanisms (a point I don't
think I've seen on this thread, although I haven't read it all).
Given a private lookup object in some patient P, a framework
class F can create a method handle on a core reflection API
point that is @CallerSensitive (like Class.forName, etc. etc.)
which allows F to call that API point with the same rights as
P, not F.

HTH
— John

Re: An alternative to "restricted keywords" + helping automatic modules

[John Rose]

On May 18, 2017, at 1:59 AM, Stephan Herrmann  
wrote:
> 
> In all posts I could not find a real reason against escaping,
> aside from aesthetics. I don't see this as sufficient motivation
> for a less-then-perfect solution.

So, by disregarding esthetics...
> 
> Clarity:
> I'm still not completely following your explanations, partly because
> of the jargon you are using. I'll leave it to Alex to decide if he
> likes the idea that JLS would have to explain terms like dotted
> production.
> 
> Compare this to just adding a few more rules to the grammar,
> where no hand-waving is needed for an explanation.
> No, I did not say that escaping is a pervasive change.
> I never said that the grammar for ordinary compilation units
> should be changed.
> If you like we only need to extend one rule for the scope of
> modular compilation units: Identifier. It can't get simpler.
> 
> 
> Completeness:
> I understand you as saying, module names cannot start with
> "transitive". Mind you, that every modifier that will be added
> to the grammar for modules in the future will cause conflicts for
> names that are now legal, and you won't have a means to resolve this.
> 
> By contrast, we can use the escaping approach even to solve one
> more problem that has been briefly touched on this list before:
> 
> Automatic modules suffer from the fact that some artifact names may
> have Java keywords in their name, which means that these artifacts
> simply cannot be used as automatic modules, right?
> Why not apply escaping also here? *Any* dot-separated sequence
> of words could be used as module name, as long as module references
> have a means to escape any keywords in that sequence.
> 
> 
> Suitability for implementation:
> As said, your proposal resolves one problem, but still IDE
> functionality suffers from restricted keywords, because scanning
> and parsing need more context information than normal.

…we obtain the freedom for IDEs to disregard abnormal
amounts of context, saving uncounted machine cycles,

> - Recovery after a syntax error will regress.

…and we make life easier for all ten writers of error recovery
functions,

> - Scanning arbitrary regions of code is not possible.

…we unleash the power of an army of grad students to study
bidirectional parsing of module files,

> Remember:
> In an IDE code with syntax errors is the norm, not an exception,
> as the IDE provides functionality to work on incomplete code.

…and ease the burdens of the thousands who must spend their
time looking at syntax errors for their broken module files.

Nope, not for me.  Give me esthetics, please.  Really.

— John

Re: Disallowing the dynamic loading of agents by default

[John Rose]

On Apr 3, 2017, at 3:36 PM, mark.reinh...@oracle.com wrote:
> 
> If developers and deployers have to to opt-in to breaking encapsulation
> on the command line then that at least makes it clear to someone trying
> to diagnose a failing system that something fishy might be going on.

It seems to me that some of the legitimate use cases use injected
code to perform similar diagnoses.  Having been many times on the
wrong side of a failing system, I am sympathetic with any diagnostic
tool, no matter how invasive.  And, yes, I give extra credit to tools
that can be applied online with no prearrangement.

This seems to lead back to allowing ad hoc code injection out
of the box.  Would it help if, although there was not an opt-in
on the command line, there was some restriction about the
nature of the injected code?  I'm thinking what we did with
JVMCI:  Opened up a deep hole into the JVM, and then
protected it by sealing it into a module.  Can we get the
module system to help us out here also?  Maybe, out of
the box, your code must be inside some particular module
in order to do monkey-patching.  Is there a way to do post
facto loading, during an emergency, into the privileged
module, of diagnostic code?  (Probably not; modules are
static.)

Anyway, I imagine there are ways for instrumentation code
to be loaded that are far more controlled and explicit than the
current state of affairs, and still won't require prearrangement.

Possibly all such ways will end up being too abusable.
But surely it is better to have a library that says "Hey,
I'm using/defining the monkey patching module", than
a random JAR that sneaks in and fires up a self-attach.

A couple of other thoughts:

1. The "execute from JAR" use case seems to want a hook
for command line options.  It's very ugly, but maybe that would
relieve pressure on users who cannot access command lines.
(But I'd much rather see problems solved by injecting a module
info file into a JAR, rather than some new command-line knobs.)

2. I think it is inevitable that we will be using the static structure
of modules, in future releases, to guide global optimizations.
Some of the hot-patching techniques people rely on today
will come into conflict with those optimizations.  The JVM is
amazing at dealing with dynamically changing configurations,
but we also need to keep the option for static optimizations open,
in the long term.  Sometimes the cost of switching from optimized
code to debug-mode must be so great that it won't be a viable
option, especially for online problem solving.

HTH
— John

Re: Disallowing the dynamic loading of agents by default

[John Rose]

On Apr 3, 2017, at 12:03 PM, Gregg Wonderly  wrote:
> 
> Alan, it is exactly this kind of comment from the team which just tears apart 
> the whole view that you might actually be considering what everyone in the 
> Java community needs. 

I think *this* comment is unfair to Alan.  I read Alan as saying
"don't assume that users can rely on an SM present".  If I'm right,
that is a far cry from tearing the community into parts.  I think you
would admit that not everyone uses SM.  So you didn't ding Alan
(who is doing really heroic work for the community) for simply
reminding us that a SM-based approach would not serve the
whole community equally.  Did you impute some other motive to him?

— John

Re: MethodHandle performance

[John Rose]

On Jan 12, 2017, at 12:29 PM, Claes Redestad  wrote:
> 
> Right, I was just looking at the micro Stephen provided me, and it does
> seem that the added cost for this case is due to invokeWithArguments
> creating a new invoker every time.

This is a good workaround, and Stephen's report is a helpful reminder
that our performance story has a sharp edge.

We cache spreaders in the case of varargs methods,
for full performance, but not for the ad hoc spreader used by MH.iWA.

We should cache them, to remove this sharp edge (or performance pothole).
There are small technical challenges to do so.  Claes and I added
some notes to the bug report; maybe someone can look into it more.

— John

Re: Invoking default methods from a Proxy's InvocationHandler in JDK9

[John Rose]

On Jan 2, 2017, at 11:33 PM, Alan Bateman <alan.bate...@oracle.com> wrote:
> 
> On 03/01/2017 07:17, Remi Forax wrote:
> 
>> I do not think that the workaround to create a Lookup object by reflection 
>> works with 9 given that java.lang.invoke is not declared as an open package.
>> 
>> John Rose has proposed to add a method to get a private Lookup object 
>> through sun.misc.Unsafe but as far as i know, this proposal goes nowhere.
>> 
> I think we got a good place with MethodHandles.privateLookupIn. It's not 
> going to allow you get a Lookup with private access to types in core modules 
> such as java.base but it may be useful for some cases.

+1

As of JDK-8130227, it is possible to use Lookup.findSpecial to gain access to 
interface default methods.

See below for an example.

— John

/*
  $ javac InvokeDefault.java && java -ea InvokeDefault
  override foo on an OverridesDefault
  default foo on an OverridesDefault
  default foo on an OverridesDefault
  $ java version "9-ea"
  Java(TM) SE Runtime Environment (build 9-ea+131)
  Java HotSpot(TM) 64-Bit Server VM (build 9-ea+131, mixed mode)
*/

import java.lang.invoke.*;
import static java.lang.invoke.MethodHandles.*;
import static java.lang.invoke.MethodType.*;

interface HasDefault {
default String foo() { return "default foo on "+this; }
}
public class InvokeDefault {
private static class OverridesDefault implements HasDefault {
public String foo() { return "override foo on "+this; }
public String toString() { return "an OverridesDefault"; }
}
public static void main(String... av) throws Throwable {
HasDefault obj = new OverridesDefault();
System.out.println(obj.foo());
// ==> override foo on an OverridesDefault
System.out.println(HasDefaultShim.MH_foo().invoke(obj));
// ==> default foo on an OverridesDefault
System.out.println(InvokeDefault.MH_foo().invoke(obj));
// ==> default foo on an OverridesDefault
}
static MethodHandle MH_foo() {
try {
Class it = HasDefault.class;
MethodHandle mh = MethodHandles.lookup().findSpecial(it, "foo", 
methodType(String.class), it);
return mh;
} catch (ReflectiveOperationException ex) {
throw new AssertionError(ex);
}
}
}
interface HasDefaultShim extends HasDefault {
static MethodHandle MH_foo() {
try {
Class it = HasDefault.class;
Class me = HasDefaultShim.class;
MethodHandle mh = MethodHandles.lookup().findSpecial(me, "foo", 
methodType(String.class), me);
assert(mh.type().parameterType(0) == me);
// force the MH to accept all implementations of the interface
mh = explicitCastArguments(mh, methodType(String.class, it));
assert(mh.type().parameterType(0) == it);
return mh;
} catch (ReflectiveOperationException ex) {
throw new AssertionError(ex);
}
}
}

Re: Lokkup.find* do not add read edges like reflection does

[John Rose]

On Dec 6, 2016, at 1:04 PM, Alan Bateman  wrote:
> 
> The reason that java.lang.invoke checks readability is because it's aligned 
> with bytecode. It would be a significant change if the access checks in 
> java.lang.invoke differed to bytecode.

The CONSTANT_MethodHandle constants very closely align with bytecode behavior, 
in the same class-file as the hypothetical bytecodes; should they respect 
read-edges?  Yes.

The Lookup.find* API parallels those constants, with as little divergence as 
possible.  Respect read-edges there?  Yes again.

Bytecode behavior alignment means that anything you can do by loading a 
classfile (or anon-class) next to the target class can also be done by a method 
handle.  Should such aux. classes respect read-edges?  Of course.

Diverging behavior between the above three access points (to bytecode 
behaviors) is possible but undesirable.  It would make refactoring operations 
and security analysis more painful.

On the other hand, there are reasons why Core Reflection ignores read-edges.  I 
suppose they are rooted in the dynamic, ad hoc, late-bound nature of reflective 
usage, as opposed to statically bound behavior attached to a particular bundle 
of bytecodes.  MH usage is often static but sometimes ad hoc.  Some MH use 
cases are:

- dynamic language, compiled call sites (S)
- dynamic language, interpreted call sites (S?)
- lambda capture (S)
- string concat (S)
- future use of indy by javac (S)
- unit tests for APIs (D?)
- optimization of Core Reflection calls (D)

Anything marked (S) is tightly coupled to a class file, *not* ad hoc, and must 
respect read-edges.  (IMO read-edges are not a security feature but a help to 
debugging unexpectedly emergent dependencies at run time.)  There are probably 
more (D) cases I'm missing.

For that last case (D) the read-edge check by Lookup can be worked around by 
performing the lookup in Core Reflection, and then using Lookup.unreflect*.  I 
think we should consider relaxing the read-edge check *just* for that case, 
allowable since Lookup.unreflect* is not precisely coupled to BC behavior.  
That would leave the Lookup.find* and CONSTANT_MH access points unchanged 
(strongly coupled to BC behavior).

I think one reason Core Reflection requires a loophole for reads-edges is that 
it does not contain any reliable mechanism for delegated access checks.  (This 
is also why the calls are slow:  The checks are done in a hacky @CS manner.)  
Since MH-using code must always keep track of the Lookup for the principal, 
that principal's reads-edges can be expected to "make sense", and a failure to 
read is something that (should be) less likely than with Core Reflection, where 
there cannot be any accurate knowledge of the principal (whose reads-edges are 
the ones that matter).

So, that's the theory that makes sense to me.  What's the practice?  :-)

— John

Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages

[John Rose]

On Nov 11, 2016, at 7:09 AM, fo...@univ-mlv.fr wrote:
> MH.invokeWithArguments takes an array of arguments but because it is 
> specified as a varargs you may think that it works like Method.invoke, but it 
> is a trap,
> it takes the receiver and the arguments altogether into the same array.

Thanks, Remi.  That should fix the problem.

We thought a little bit about adding more overloadings to invokeWithArguments,
such as one that works like Method.invoke (one prepended argument).
The general case would be making invokeWithArguments be signature-polymorphic,
with an on-the-fly asSpreader transform on the way through.

But, such extra generality would simplify only a few use cases, and on the
other hand it would probably create plenty of confusion whenever the target
method is *also* a varargs method.

— John

Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages

[John Rose]

On Nov 1, 2016, at 12:02 PM, Andrew Dinn  wrote:
> 
> I did actually suggest a way of avoiding the use of Unsafe. You give
> your nominated module full reflective access to java.lang.invoke
> allowing it to create Lookup instances (it can actually just create a
> single all privileges lookup and use this to clone others). You don't
> need to insert a class into java.lang.invoke to do this. You simply add
> one exports directive on the command line.

The weak link in this otherwise robust scheme is the use of a command line 
option to break into the jli package. Breaking in by the back door is awkward 
and might not deliver the desired Lookup if jli code changes. And it probably 
will, over time. Using the break-in is a good Proof of concept but the finished 
product needs to use a real API provided by the JDK, and using a wormhole 
Lookup actually provided by a cooperating jli. I look forward to such a thing 
in a future JDK. 

Perhaps the right surface shape for moderate reflection is Maurizios 
"reflection manifesto" API, which is completely interface driven. This means a 
meta-CE with full deep access could mock up a "reflection manifesto" 
implementation which would expose exactly the right moderate surface area, as 
negotiated with the target module. Just an idea…

– John

Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages

[John Rose]

On Nov 1, 2016, at 10:22 AM, Jochen Theodorou  wrote:
> 
> Can we clarify "privileged code"? Privileged like in a SecurityManager in a 
> PrivilegedAction for example, for privileged like only jdk internal code? 
> Just to see it black on white ;)

Good question:  I mean the basic JDK platform implementation.  Something deep 
in java.base.  Like Unsafe.

Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages

[John Rose]

On Nov 1, 2016, at 9:53 AM, David M. Lloyd  wrote:
> 
> 1. It requires the target class to be initialized
> 2. It requires the target class to proactively donate MethodHandles or a 
> Lookup to the lookup class

Both of these can be overcome, though only by privileged code.
The privileged code would forge (uh, "mint") a legitimate Lookup to the 
not-yet-initialized class.
A "Vault" meta-framework doesn't need to inject a Lookup donation statement 
into anybody's .
Instead, it needs to do the super-user operation of making a trusted lookup.
It must also fulfill the super-user *responsibility* of not leaking such 
lookups, just using them in a predictable, rule-based manner.
— John

Re: New proposal for #ReflectiveAccessToNonExportedTypes: Open modules & open packages

[John Rose]

On Oct 27, 2016, at 9:07 AM, mark.reinh...@oracle.com wrote:
> 
> Further comments most welcome, as usual!


+100 for *qualified* opens; this puts an important limit on deep reflection.

My main concern with reflection is to avoid "falling off the encapsulation 
cliff"
the first time a user module wishes to open itself up for deep reflection.
By such a "cliff" I mean that when deep reflection is allowed, potentially
any name in the module can be inspected.  Even of only 0.01% of names
are actually inspected, tools for reorganizing code must assume that 100%
of the names *might* be inspected, at some point in the future, by the
allowed party.

So qualified opens is a partial solution.  And, I think it is exactly right for
today, because it can be extended tomorrow.  I.e., it is future-friendly.
I'll explain by sketching a concept called "moderate reflection", or "MR".

A fuller solution would allow other tools to make conclusions about more
specific limitations on the actual extent of the deep reflection.   Call such
an extent-limited reflection moderate reflection, where "moderation" is
declared statically by some sort of flexible declarative rule set.  In this way
tools could make more detailed conclusions about the encapsulation of
particular names in the module.

 For future JDK releases we can consider layering  of this qualified opens,
by creating a trusted, parameterized meta-framework ("Moderate Reflection")
to which a user module can be qualified-open.  This meta-framework can
then perform deep reflection on behalf of self-describing "moderately 
reflecting"
client frameworks.  User modules would not directly open themselves to any
module other than MR.

The advantage would come when the client frameworks explicitly declare their
self-restraint ("moderation") in reflection.  For example, the "Foo" framework
might limit itself to names annotated with @FooFrameworkHook.

An AOT compiler (or other jlink-time reorganization tool) would (1) note that
some user module bar is qualified-open only to Moderate Reflection, and
has an export (maybe qualified) to a Foo framework.  At runtime, (2) the
Foo framework would ask Moderate Reflection to reflect into bar.
Moderate Reflection would (3) check the self-limitation on Foo, and also
ensure that bar exports (though doesn't open) to Foo.  MR would then
gain the requested access and delegate it back to Foo.  Finally, (4) the
AOT compiler would free itself to omit metadata for unexported names
not visible to moderate reflection.

(Method handles are a good way to grant access, since they can
be delegated from MR to Foo without re-authorizing Foo.  Core
Reflection re-authorizes on every access, which awkwardly
requires deep reflection on usage, as well as initial lookup.)

Moderate Reflection might also look at metadata on the bar module
(besides its module exports) to impose further limits on reflection.
Thus, MR-specific metadata might prevent MR from granting bar
access to normally-trusted frameworks that bar doesn't trust, if
bar can blacklist them for itself in a place that MR checks.

And so on.  None of this needs to be built into the module system.
To gain the benefit of it, basic parametric policies need to be implemented,
adopted by client frameworks, and used by tools to perform optimizations.

A typical optimization is "tree shaking", where a class or method that is
never used is dropped from the application.  Or, if it is used in a way that
can be inlined locally, the metadata can still be dropped.  In either case,
making code go dead is an important ingredient to many optimizations.

— John

Re: Alternative mechanism for reflective access control (#ReflectiveAccessToNonExportedTypes / #AwkwardStrongEncapsulation)

[John Rose]

On Sep 29, 2016, at 2:29 AM, Remi Forax  wrote:
> 
> This bug only talk about the callee side, i.e. accessing to a non accessible 
> class by a framework.
> The other part, the caller side, is to have a way to emit a call that send 
> the caller lookup to a framework,
> the easy way to do that is to have javac to emit an invokedynamic instead of 
> an invokeinterface/invokevirtual
> after having verified that everything is typechecked.

FTR, I added a comment to the bug that expands on this theme.

https://bugs.openjdk.java.net/browse/JDK-8162494?focusedCommentId=14010672 


— John

P.S.  CC-ed here:

Remi Forax notes that this proposal is about gaining access to classes.
http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009518.html

Once somebody (the class itself, a neighboring class, a module, a layer, the 
Unsafe API) grants you a Lookup object, you can call any class feature the 
lookup can access, by using a method handle.

The most general Lookup object, for a given class, can always be created by a 
method in the class's bytecodes that looks like this:

class C {
   private C() { } protected void m() { } int f; // etc., etc.
   public MethodHandles.Lookup grantAccess()  { return MethodHandles.lookup(); }
}

In this example, the "grantAccess" method gives the whole world (or at least 
the module) full access to all class members, up to and including privates.

The essence of the present proposal is to introduce mechanisms for creating 
Lookup objects which (a ) can be as powerful (on class C) as that returned by 
the "grantAccess" method above, and (b ) do not require any changes to the 
bytecodes of the class itself.

The difficult part of the proposal is designing the alternative Lookup factory 
in such a way that it doesn't just "give away the store" (as presumably the 
Unsafe version would).  The Lookup object should be (c ) handed out by a 
trusted delegate of C (such as its module layer), and (d ) only handed to 
callers who have been vetted in some way (such as by a SecurityManager check, 
as by setAccessible).  Finally, the Lookup object must be (e ) right-sized, so 
that it gives only access (say) to public methods, or package-private methods, 
according to the needs of the caller and the rules of the factory.

Another possible framework-like operation that Remi brings up is creating 
simulations of calls from managed framework code to other frameworks.  In some 
cases, this can also be done with the Lookup API.  If the callee is 
caller-sensitive (like Class.forName), the Lookup API handles it specially, 
embedding the Lookup's target class into the method handle it returns (for that 
callee) in such a way that when the method handle the caller-sensitive method 
observes the Lookup's target class, not the current invoker of the method 
handle.  Only the immediate caller is "edited" in this way; a full stack walk 
will see the specially-embedded caller and all of frames of the current 
invocation.

Another use case for frameworks is enumerating and examining (e.g., annotations 
of) reflected class members, instead of just invoking them.  The Lookup API is 
intended to help with this also.  The Lookup.revealDirect method can translate 
a previously accessed method handle into a Core Reflection object that 
describes the same underlying class member.  The Lookup API (as of 9) also has 
the ability to resolve class names with respect to the Lookup's target class.  
Finally (as noted above) calls to caller-sensitive Core Reflection API points 
can be made to appear originating from a target class by looking up those API 
points via a full-power Lookup on the target class, instead of calling from 
normal bytecodes.

Personally, I would like to encourage people interested in structured access 
elevation, by bona fide frameworks, to propose securable mechanisms for handing 
out Lookup objects, or similar restrictable capability objects.

JDK-8162494: some frameworks need access delegation

[John Rose]

On Jul 24, 2016, at 1:24 AM, Alan Bateman <alan.bate...@oracle.com> wrote:
> 
> On 23/07/2016 00:12, John Rose wrote:
> 
>> 
>> The Lookup API does not have a "setAccessible" analog.
>> Maybe it should; along the lines of Unsafe.privateLookup(Class).
>> 
> Would this be the internal Unsafe or sun.misc.Unsafe? I just wonder if this 
> is something that would be regretted in the future.

It has to be designed carefully.  I've filed a public tracking bug, which folks 
can use as a point of reference.
  https://bugs.openjdk.java.net/browse/JDK-8162494
  some frameworks need access delegation via MethodHandles.lookup

The regrets come from two circumstances:  Leaving the store open after hours,
or requiring premium customers to break in the back door for legitimate 
purchases.

> For the original issue then maybe we should look again at providing a way to 
> invoke a default method from a proxy's invocation handler. Peter Levart had a 
> reasonable proposal on that. It would of course require a lot of security 
> eyes to make sure that it is bullet proof.

Yes, I've wandered pretty far away from that issue.  (Changed the subject line!)

— John

Re: creating proxies for interfaces with default methods

[John Rose]

On May 27, 2016, at 1:16 AM, Alan Bateman  wrote:
> 
> I don't think it's possible to create a Lookup via support API with just the 
> PRIVATE lookup mode. It only seems to work because this code seems to hack 
> into the non-public constructor. I'm curious if you invoke toString on this, 
> with JDK 8 and with -esa, as I assume you will get a similar assertion.

The Lookup API does not have a "setAccessible" analog.
Maybe it should; along the lines of Unsafe.privateLookup(Class).
— John

Re: Should setAccessible be part of Java or not? (was Re: It's not too late for access control)

[John Rose]

Thanks, Andrew, for a good summary of the practicalities behind
access overrides.

On Jul 18, 2016, at 2:34 AM, Andrew Haley  wrote:
> 
> The wider the access, the more scope for complex interfaces.
> 
> Also, making a method or field globally accessible may mean that the
> author of the code can no longer guarantee its correctness.
> 
> … we can reduce risk and the size of attack surfaces

I especially agree with these parts, from bitter experience.  When you
widen an interface, you add attack surface.  If you widen an interface
with clear documentation and buy-in from the author of the interface,
and can add both negative and positive tests for the new API surface,
then you can sleep at night.  Meanwhile, the best attack surfaces are
implicitly defined (e.g., privates which aren't really private after all)
and are activated by remote control (e.g., setA), without the buy-in
of the library author.

I'm not saying this to condemn frameworks which perform polymorphic,
pattern-driven access overrides, but to explain their special risks.  And
I think in the future we will have a much more explicit and robust set
of rules for opt-in and opt-out into such frameworks, including optioning
which is delegated to third parties, yet is still limited to a small subset
of what is available to root privileges (aka. setA).

Finally, as JIT guy, I'd like to note that one way to crisply evaluate
the "weight" of an API surface given away by some "fancy feature"
(ubiquitous override-ables, dynamic loading, monitor-per-object, AOP,
access overrides, …) is to see how many optimizations it makes difficult
or impossible.  Yes, Java is wonderfully plastic, and yes, we need to
supply hardening transformations so the plastic can be made durable.

— John

Re: Should setAccessible be part of Java or not? (was Re: It's not too late for access control)

[John Rose]

On Jul 14, 2016, at 3:25 PM, Jason Greene  wrote:
> 
> Not to sound like a broken record, but not all systems want the module to 
> control its own security. They want an intermediary.
> 
> And that's a perfectly fine and reasonable security model.


I agree, as long as the intermediary is highly predictable and accountable
in its violations of standard access rules.

And that's in the middle ground I'm talking about, where a module (or class)
is deemed (without direct delegation on its part) to extend trusted access to 
some
intermediary, which is generally trusted to do things on behalf of a broad 
range of
such modules (or classes).  It's middle ground (not the extreme of setA) because
(a) the intermediary would be more clearly marked as trusted (not just contain
a setA call somewhere in its guts), and (b) the task entrusted to the 
intermediary
would not require trusted access to all classes, but some well-defined "slice"
across a well-defined set of classes.

To reason about setA you have to predict the actions of a Turing machine,
as it assembles Class and String constants to pick out any API point whatever,
and then make a skeleton key for it.

A good framework makes those actions predictable, based on rules which
allow predictions based on the form of a program rather than the steps it
takes during execution.

Personally, I prefer the MethodHandles.Lookup API to setA (when I have
a choice between the two) because Lookup respects the existing access
control boundaries, while allowing delegation.  I think the data flow of
capabilities is easier to reason about than the data flow of random String
and Class values, followed by an unpredictable setA call.  But a Lookup
(today) must be created by the principal that owns the access rights,
which is what I think Jason is objecting to.

If we were to make some sort of middle ground between setA and Lookup,
it would look like a kind of Lookup that was obtained with upgraded access
(relative to publicLookup), but without direct creation by the target class.

In exchange for that power, there would be rules for limiting and configuring
the effect of such a Lookup.  For example, it might work only for API points
carrying a particular annotation.

You can build this sort of thing on top of setA of course, since setA is
basically root privileges.  My point is not that access rules should be
inviolate, but that exceptions to the rules should be as predictable and
even configurable as possible.

A final point:  Predictability and configurability leads to alternative
implementations, such as AOP-like injection of code at AOT time.
That's one of the promises of jlink.

HTH
— John

Re: Should setAccessible be part of Java or not? (was Re: It's not too late for access control)

[John Rose]

On Jul 14, 2016, at 4:51 AM, Andrew Haley  wrote:
> 
> Forgive me if I've missed something, but
> #ReflectiveAccessToNonExportedTypes does not deal with the need to
> make fields or methods accessible to the framework.  That's what
> setAccessible is used for.  It would certainly be nice for a
> framework to be able to say "make it accessible, but only to me."

Saying setAccessible is like "borrowing" (without owner permission) a
key to one locked door, if a non-public method is like a locked door.

Today's MethodHandles.Lookup object gives another way to open such
doors.  But you have to obtain the lookup object from a party that already
has access rights.  It is like the owner of a building (a class) giving a key
which opens all the doors in the building, or all the doors not marked 
"Private".

With both setA Methods and Lookups, once you have the key in hand,
you have to lock it up to prevent bad guys from stealing it from you.
And if you loan it out, you have to loan it to trustworthy parties.

Somewhere in between the two (unrestricted "borrowing" vs. direct delegation
of original access rights) must be some better conventions for reflecting into 
frameworks.

— John

Re: It's not too late for access control

[John Rose]

On Jul 12, 2016, at 9:38 AM, Andrew Dinn  wrote:
> 
> I'm not sure how AOT compiled code is supposed to protect itself against
> the potential consequences of my agent tweaking accessibility in this
> way. Would it be not require -- at the least -- wholesale deoptimization
> of compiled methods in the module to which the package is exported? How
> would one arrange for JITted code to trap out and invalidate any
> speculative optimizations that might be invalidated by this incremental
> export?

Either wholesale de-opt, or a fast-fail error when the injection is attempted.
(It's another example of the tug-of-war between parties with different views on 
a class.)

Either outcome is bad enough that we need speed bumps to prevent people
from just cruising heedlessly into the failure zone.  By "speed bump" I mean
a need to assert extra-specific intention, and/or obtain an access right,
in order to increase the visibility of (or inject code into) a module that
has been consolidated into an AOT sub-assembly.  (N.B. I'm using these
terms in a general sense, not referring to any particular tech.)

BTW, if you contrive to inject a MethodHandles.Lookup donor into
a class, the donee can get everything (99.9%) the class can do by
materializing and calling appropriate method handles.  With a little
care (bind the MHs to static finals) you get identical performance.  
This is easier than spinning adapters and injecting those one by one.
Is that how you are doing your injection?

— John

Re: It's not too late for access control

[John Rose]

On Jul 12, 2016, at 9:00 AM, John Rose <john.r.r...@oracle.com> wrote:
> 
> lurk mode

P.S. I forgot one more point:  As JVM/JIT guy I find the prospect of sealing
packages very appealing.  Maybe it's a false promise, but it would be a
home run if we could reliably seal sub-assemblies for tree-shaking AOT
compilation.  Doing this requires pretty strong assurances that the AOT
engine can enumerate all uses of a "public" class.  Doesn't this use case
interfere with the proposed principle that "public always means global"?
Again, it's a tug-of-war between the class author and an assembly-time
operation.

Re: It's not too late for access control

[John Rose]

On Jul 12, 2016, at 8:01 AM, Paul Benedict  wrote:
> 
> All things being equal with Jigsaw features today, I'd rather have "public"
> retain it's global visibility and extend "package private" to the module.

Again, $0.02, with apologies in advance for opining on a subject I'm not
an expert in.

From a pure language POV I find this appealing, but in our world there
are myriads of existing public classes already coded in libraries which need
large-scale encapsulation.  Surely we need to be able to tell some libraries,
"Hey, not so public, there; we're restricting your visibility."  If we can't do 
that
we don't have modules at all.  If Java had modules from the start, maybe
we'd say "public is globally public, so any class can punch through all layers 
of
encapsulation by using the super-strong keyword 'public', and the default
is 'module'."  (Advance apology if this is a straw-man.)

> This makes great sense to me. Why? Because when I develop a library, I
> really don't see any great benefit in hiding a type from another package.
> Let me qualify that I really don't see any great benefit in hiding a
> type from another package **that I own!** (double emphasis). I have
> inspected all my uses of "package private" and it always comes down to one
> thing: preventing static linkage to the type from packages **outside my
> library**.

To me this looks like a simple power struggle between the class author
expecting a platform for full publicity, and the system assembler expecting
to be able to control inter-library visibility.  Both can't have the final word.
Shouldn't the assembler have it?  And if the assembler is given absolute
control over some unit of access control, it seems likely to me that this
unit has to look a lot like a Jigsaw module.

Note that system assemblers cannot be expected to edit individual classes.
That would be another scale error.  So in the end, the author of a class and
of a package has to provide a provisional, local spec. of what's visible, while
the assembler (who works with unmodifiable libraries) owns the global spec.

(HTH.  I'm probably missing a couple of Third Way options; please point them
out.  I'll go back to lurk mode now.)

— John

Re: It's not too late for access control

[John Rose]

On Jul 12, 2016, at 2:00 AM, Remi Forax  wrote:
> 
> First, there is already in the Java ecosystem a notion of non-exported 
> package, packages startings with com.sun or packages containing internal, it 
> was just a convention and not something enforced by the VM. What the JPMS 
> spec does is just to normalize how to declare an exported package and mandate 
> that the VM check this new rule.
> 
> Sure it means that public classes are not accessible/visible by everybody 
> anymore, but a class like sun.misc.Unsafe was never really accessible by 
> everybody despite being declared public.
> 
> Furthermore, declaring if something is exported or not at class level instead 
> of at package level seems wrong to me, usually, several classes works 
> together for a purpose and you want these classes to be exported or not, so 
> it's not something that should be decided at class level.
> 
> So i see the JPMS spec conept of non-exported package as a standardization of 
> an existing practice not something new that people will have trouble to 
> understand and reason about.

My $0.02:

As an author of Unsafe I agree enthusiastically with this viewpoint.  (Well 
stated, Remi.)  System assemblies (not individual classes or packages) need to 
isolate portion A of the global package namespace from portion B.

For me, declaring such assembly instructions inside class definitions would be 
a scale error.  The access control in question is not a relation between 
individual classes; it's a relation between larger entities: packages.

It's a relief to see the old ad hoc package-scale hiding mechanisms (restricted 
packages, shared secrets, providers, class loaders, etc., etc.) getting 
supplanted by a first-class formalism.  It's also scary to see the amount of 
work involved, but it's worth it.

— John

Re: RFR(S): 8160564: TEST: Add a test to check the implementation of VersionProps.versionNumbers()

[John Rose]

On Jul 7, 2016, at 11:05 AM, Martin Buchholz  wrote:

> private static final jdk.internal.misc.Unsafe 
> java.util.concurrent.ConcurrentHashMap.U

Unless the security manager is turned on, you can do setAcc to pick up all 
sorts of private fields.

As Alan points out, it would be good to tighten this up, but that will require 
new logic which somehow avoids breaking existing legitimate uses.

(The new logic can't be some hack like "check for Unsafe".  Point-fixes like 
that just complicate the security model, creating dark corners for black hats 
to work in.)

The JVM and language are loose about mentioning inaccessible (or even 
non-existent) classes in field and method descriptors.
In particular, the JVM does not resolve or access-check names which occur as 
types of fields, parameters, or return values.
So accessing a field of type "Unsafe" (or even "NoSuchType") gets a pass, as 
long as the privacy of the field is somehow taken into account (via setAcc, 
here).

> jdk.internal.misc.Unsafe@35851384

Here we get an object of a non-exported type.  That happens all the time; an 
anonymous inner class or lambda expression does this.

Again, the language and JVM are loose about letting objects of internal type 
"escape" via public APIs.  Arguably this is fundamental to OOP.

> class jdk.internal.misc.Unsafe

Here we inspect the class of the mysterious escaped object.  You can do this to 
AICs and lambdas too.

Our legacy reflection system says that Object.getClass returns the actual class 
of the object, without access check.
Again, that's loose, and difficult to tighten compatibly with existing 
applications.
In particular, adding an access check to getClass (which is the obvious move 
here) will perturb the performance model.
Currently, getClass requires one or two memory references.  An access check is 
likely to more than double that, with noticeable effects on dynamic languages.

Nothing here is impossible to fix, but everything is tricky, because of 
backward compatibility.

> Exception in thread "main" java.lang.IllegalAccessError: class GetUnsafe9
> (in unnamed module @0x649d209a) cannot access class
> jdk.internal.misc.Unsafe (in module java.base) because module java.base
> does not export jdk.internal.misc to unnamed module @0x649d209a
> at GetUnsafe9.main(GetUnsafe9.java:16)

Finally we get an error.  The specific error comes from an invisible cast which 
follows a successful call to the Class.cast method:

   //jdk.internal.misc.Unsafe U = unsafeClass.cast(theUnsafe);
   // ==>
   Object $tem = unsafeClass.cast(theUnsafe);  // descriptor is (Object)Object
   jdk.internal.misc.Unsafe U = (jdk.internal.misc.Unsafe) $tem; // javac 
inserts this cast

The "checkcast" instruction in the second statement fails because it must 
resolve the type Unsafe.

Even if there were other ways to get your hands on a well-typed Unsafe 
instance, as soon as you attempted to resolve an "invokevirtual" against it, 
you'd hit the same class resolution error as "checkcast" did.

The reflection API performs similar checks to the source compiler and JVM 
bytecode executor (verifier & link resolver).
However, since it works on "live" class pointers there is no resolution step to 
catch "bad" class references.
So Class.cast and Method.invoke either have to let the "bad" class operate 
(which Class.cast does) or perform an additional ad hoc access check (which 
Method.invoke does).
The additional access check is useful, but setAccessible nullifies it.
Paradoxically, Class.cast (which doesn't have the ad hoc check) is more 
restrictive than Method.invoke, since javac puts in the invisible cast (forcing 
resolution on Class.cast) while Method.invoke can be neutered by setAccessible.

Again, we are back to finding ways to rein in setAccessible.

HTH
— John

Re: RFR: 8152641: Plugin to generate BMH$Species classes ahead-of-time

[John Rose]

I like it too.  Reviewed.

A couple of comments:

The default list of species in the plugin (List.of("LL", "L3", … "I", "LLILL")) 
should
be commented with information for maintainers to reproduce the procedure you
used to generate that list.  Otherwise the list is just magic, and 
unmaintainable.

I'm glad to see this pre-generation work happening.  Please consider (or file
a followup RFE) pre-generating other classes dynamically generated by jli.
The lambda form editor produces a large number of stereotyped classes.
In the common case where an anonymous class is generated just for one
static method, the linker plugin should generate a *single* named class
containing *all* of the static methods.

Eventually, I think, you will want a jlink option which will control the 
pre-generation
of all sorts of jli classes.  Perhaps it would be it would be something like
"--generate-jli-classes=", where the argument includes a way
to specify all the BMH species you are working with now, but would also
provide a way to ask for LFE transforms.  What you have now is fine as
far as it goes.

— John

On Mar 30, 2016, at 10:11 AM, Vladimir Ivanov  
wrote:
> 
> Nice work, Claes!
> 
> j.l.i part looks fine.
> 
> Best regards,
> Vladimir Ivanov
> 
> On 3/30/16 7:17 PM, Claes Redestad wrote:
>> Hi Peter,
>> 
>> something like this, then:
>> 
>> http://cr.openjdk.java.net/~redestad/8152641/webrev.05/
>> 
>> - Added a method only used by the plugin which validates input; added a
>> comment about the dependency
>> - Invalid types are logged but otherwise ignored now (I bet someone
>> might suggest a better way to handle user errors)
>> - Some cleanup, introduced constant for class name prefix and removed
>> duplicate type string shortening etc
>> 
>> /Claes
>> 
>> On 2016-03-30 17:17, Peter Levart wrote:
>>> Hi Claes,
>>> 
>>> webrev.04 looks good now.
>>> 
>>> Maybe just one nit. For production quality, plugin input could be
>>> verified that after expansion it is composed of just the following
>>> characters: "LIJFD". Otherwise ClassWriter might generate an unusable
>>> class without complaining (for example if someone sneaks-in characters
>>> 'S' 'Z' 'B' or 'C')...
>>> 
>>> Or, better yet, create another method in BMH that will be the "public"
>>> API between the plugin and BMH which does the validation and calls
>>> generateConcreteBMHClassBytes(). Internally in BMH the validation is
>>> not necessary as the types strings are composed programmatically and
>>> are guaranteed to be correct.
>>> 
>>> Regards, Peter
>>> 
>>> On 03/30/2016 04:15 PM, Claes Redestad wrote:
 
 
 On 2016-03-30 14:21, Peter Levart wrote:
> Hi Claes,
> 
> On 03/30/2016 12:53 PM, Claes Redestad wrote:
>> Hi,
>> 
>> I think Class.forName(Module, String) seemed like a nice
>> efficiency/simplicity compromise, but there is some usage of
>> lambdas/method-refs in the Module lookup code, especially for
>> exploded modules (which get exercised during JDK build). Depending
>> on a lambda from code in java.lang.invoke means we fail to bootstrap.
>> 
>> But hey, we're living in an encapsulated world now, and this is in
>> java.base, so why not go directly to the BootLoader:
>> 
>> http://cr.openjdk.java.net/~redestad/8152641/webrev.03/
>> 
>> Since this is what's called from Class.forName(Module, String), the
>> overhead should be even smaller than in your classForNameInModule
>> test.
> 
> Good idea.
> 
>> 
>> If I call this final, will you say "Reviewed"? :-)
> 
> Sorry, I don't posses the powers. :-)
> 
> I could say "rEVIEWED", but...
> 
> In the plugin, the input is shortened speciesTypes strings. What
> guarantees that they really are normalized? For example, If one
> specifies "LLL" as input, it will get expanded into "LLL", the
> generated class will have "_L3" as a name suffix, but you will pack
> it in the image with "_LLL.class" filename suffix.
> 
> That's another reason why a method in BoundMethodHandle$Factory with
> the following signature:
> 
> Map.Entry generateConcreteBMHClassBytes(String types);
> 
> ...would be a good idea. It would return class bytes and the name of
> the class which you could use to derive the class filename without
> hardcoding the same logic in plugin and in BMH.
> 
> You just move the "LambdaForm.shortenSignature(types)" from
> getConcreteBMHClass and className/sourceFile calculation from
> generateConcreteBMHClass down to generateConcreteBMHClassBytes
> method and change the signatures...
 
 Yes, it makes sense to keep control over the class name inside the
 factory class, and this does allow specifying shortened or expanded
 forms (L3 vs LLL) interchangeably as input to the plugin, which
 reduces possibility for user errors.
 
 How about this: 

Re: MethodHandles.Lookup and modules

[John Rose]

On Dec 17, 2015, at 6:01 PM, John Rose <john.r.r...@oracle.com> wrote:
> 
> I think I would prefer case 2.  The user model is PUBLIC is the weakest 
> (non-empty) access
> mode available to bytecode behaviors.  As such it respects the LC's position 
> in the module
> graph, and excludes module-private, package-private, and class-private.  
> UNCONDITIONAL
> is the special thing provided by publicLookup, which ignores the module 
> graph.  Then
> PACKAGE opens up the LC's package, MODULE opens up the LC's module, and 
> PRIVATE
> opens up the LC itself (plus its nestmates).  Feels pretty good, especially 
> since MODULE
> and PACKAGE continue to have a parallel sense of restriction.
> 
> What do you think?

So I caught you in the hall and we talked, and this seems agreeable to us both,
perhaps with a name change to UNCONDITIONAL, and also a distinction between
PUBLIC and QUALIFIED (as you originally proposed).

To try and tease out some symmetry here:
- Always, any type T is accessible to itself, when T = LC.
- PACKAGE mode: Any type T is accessible (within its own package), when 
PACKAGE(T) = PACKAGE(LC).
- MODULE mode: A public type T is accessible (within or beyond its package), 
when MODULE(T) = MODULE(LC).
- QUALIFIED mode: A public type T is accessible beyond its module, when 
IS_CE(T, LC),
where IS_CE(T, LC) = IS_CONDITIONALLY_EXPORTED(PACKAGE(T), MODULE(LC)) and 
MODULE(LC) READS MODULE(T).
- PUBLIC mode: A public type T is accessible beyond its module friends when 
IS_UE(T, LC),
where IS_UE(T, LC) = IS_UNCONDITIONALLY_EXPORTED(PACKAGE(T)) and MODULE(LC) 
READS MODULE(T).

These conditions can be tested independently.  PACKAGE implies MODULE, but 
everything else is disjoint.

Also:
- UNCONDITIONAL: In this mode, a type T is accessible if 
IS_UNCONDITIONALLY_EXPORTED(PACKAGE(T)), regardless of LC.
- PRIVATE/PROTECTED: These protection modes apply only to non-types (JVM does 
not enforce "private" on classes).
- NOACCESS: This is not a mode but the absence of any combination of modes; no 
access is allowed.

The publicLookup should have UNCONDITIONAL and PUBLIC set.
An original full-power lookup does *not* have UNCONDITIONAL set, just PUBLIC.
The purpose of UNCONDITIONAL is to allow publicLookup to be unconcerned
(as documented) about its LC.  We can restore LC to be java.lang.Object.

The distinction between QUALIFIED and PUBLIC is present simply because of
the logical fact (as you point out) that, if you teleport to a new module, you
must lose your qualified imports, but you shouldn't lose your unconditional 
ones.

The distinction between PUBLIC and UNCONDITIONAL is present in order
to capture the differing behaviors of lookups derived from publicLookup and
those derived from full-power lookups.

The presence of MODULE captures the larger but package-like scope of
a module's internal names.

About "mode stripping":

You suggested (which sounds OK) that there is no need to "validate" bit masks 
of lookup objects.
Just have excludeModes clear some bits and continue.  This means there can be 
lookups which
are able to read (say) using PACKAGE mode but not PUBLIC mode.  (Today's 
lookups can
have PUBLIC without PACKAGE but not vice versa.)  Each mode bit enables a single
line in the above logic, and (as you can see) the lines can be applied 
independently.
Some implications follow.

When looking up a type member T.M, the additional access checking on the
member's enclosing type T (from LC) may lack PACKAGE, MODULE, QUALIFIED,
and PUBLIC modes (if excludeModes stripped those bits from a full-power lookup).
This means that such mode-stripped lookups can (logically) only obtain members
LC.M from the single lookup class LC (since a class always has access to 
itself).
That seems reasonable and useful.

A stripped lookup with only MODULE or QUALIFIED bits will never be able to "see"
any T.M, since members only match in PUBLIC, PRIVATE, PROTECTED, and
PACKAGE modes.  Odd, and probably not useful, but logical.

There does not seem to be a way to say "give me only the T.M for which T
is package-private and M is public", or "give me only the T.M for which T
is module-private but M is public".  Those can be composed on top,
especially with the new Lookup.accessClass API point as a post-test on T.

As an odd use case, a stripped lookup with only PACKAGE modes
will be able to see any package-mate T of LC, and any package-private
API point T.M, but it won't be able to query anything *outside* of the
package of T.  Unfortunately, it also won't be able to query any public
member T.M, unless the PUBLIC bit is present.  So I suppose stripping
MODULE and QUALIFIED, leaving PUBLIC and PACKAGE, would
provide useful access to T.M even if M were public.

As you see, I'm trying to apply the same mode bits to both types
and their members, as mechanically as possible.  One place where
that is tricky

Re: MethodHandles.Lookup and modules

[John Rose]

> On Dec 18, 2015, at 4:44 PM, Alex Buckley  wrote:
...
> So let's recap full power lookups:
> 
> - Start with an arbitrary class in an arbitrary module calling 
> MethodHandles.Lookup.lookup() to get a "full power" lookup object L. L's 
> lookup modes are PUBLIC + QUALIFIED + MODULE + PROTECTED + PACKAGE + PRIVATE.
> 
> - The arbitrary class obtains a Class object representing class A, then calls 
> L.in(A). If L's lookup class cannot access A (for example, A is 
> package-private in a different package than L's lookup class), then the 
> resulting lookup object has lookup mode NOACCESS. Otherwise:
> 
>  -- If A is in a different module than L's lookup class, then the resulting 
> lookup object has lookup mode NOACCESS.
> 
>  -- If A is in the same module as L's lookup class, but a different package, 
> then the resulting lookup object has lookup modes no greater than PUBLIC + 
> QUALIFIED + MODULE.
> 
>  -- If A is in the same module as L's lookup class, and in the same package, 
> but A is a different class than L's lookup class, then the resulting lookup 
> object has lookup modes no greater than PUBLIC + QUALIFIED + MODULE + PACKAGE.
> 
>  -- If A is nested in the same package member as L's lookup class, then the 
> resulting lookup object has lookup modes no greater than PUBLIC + QUALIFIED + 
> MODULE + PACKAGE + PRIVATE.
> 
>  -- If A is the same class as L's lookup class, then the resulting lookup 
> object has the same lookup modes as L.

I agree; that's just how they teleport.

> 
>> The publicLookup should have UNCONDITIONAL and PUBLIC set. An
>> original full-power lookup does *not* have UNCONDITIONAL set, just
>> PUBLIC. The purpose of UNCONDITIONAL is to allow publicLookup to be
>> unconcerned (as documented) about its LC. We can restore LC to be
>> java.lang.Object.
> 
> Since PUBLIC is just UNCONDITIONAL with a concern for readability, it's 
> surprising that publicLookup cares about PUBLIC.

The PUBLIC bit is along for the ride.  It's meaningless when accompanied by 
UNCONDITIONAL, and can stay on, meaningfully, after UNCONDITIONAL drops off.

(Actually, PUBLIC is also required when you are looking up a *member* C.M of 
class C using publicLookup.  Both C and C.M had better be marked "public".  And 
you had better have the PUBLIC mode bit set so you can make use of that fact.  
The C.M is not marked "unconditional".)

> Before saying any more about that, let me take a small detour. I seem to 
> recall an intent to specify the public lookup object as representing an 
> (undisclosed) lookup class in the unnamed module ... if so, then since the 
> unnamed module reads all named modules by decree, we have 
> UNCONDITIONAL+PUBLIC as trivially equal in access power to PUBLIC, and we 
> don't need UNCONDITIONAL at all ... if not, then publicLookup could be 
> UNCONDITIONAL only.

True; this uses that odd feature of the unnamed module.  It's clever, but 
surprising.

With UNCONDITIONAL, though, we don't need to use that feature, and can 
compatibly retain the JDK7/8 behavior of PL.LC == Object.

Also, without UNCONDITIONAL, you can't teleport publicLookup to another module 
(you drop to NOACCESS).

That is a mildly bad thing, because Lookup.findClass assumes flexible 
teleportation.

The use case with PL is:
Q: I want to find an unconditionally readable class named "Foo", from the 
viewpoint (classloader & module) of the class Bar whose Class object Bar.class 
I possess.  How?
A: publicLookup().in(Bar.class).findClass("Foo").

Basically, adding UNCONDITIONAL allows Lookup to manage the magic directly, 
instead of leaning on the unnamed module, with more power to boot.

UNCONDITIONAL also tracks the fact that the Lookup object came from 
publicLookup, not full-power-lookup.  If not a mode bit, how else will be do it?

> Returning from the detour ... does the public lookup object have PUBLIC 
> solely so that it can teleport to give new lookup objects which drop 
> UNCONDITIONAL but still retain the interesting PUBLIC mode?

Yes; see above for an example.

> Proceeding to walk through a publicLookup:
> 
> - Start with an arbitrary class in an arbitrary module calling 
> MethodHandles.Lookup.publicLookup() to get a public lookup object PL. PL's 
> lookup mode is UNCONDITIONAL [+ PUBLIC?].

Yes, UNCONDITIONAL + PUBLIC.

> - The arbitrary class obtains a Class object representing class A, then calls 
> PL.in(A). If PL's lookup class cannot access A [I'm not sure what PL's lookup 
> class is, but it seems plausible that it can't access A], then the resulting 
> lookup object has lookup mode NOACCESS.

If PACKAGE(A) is unconditionally exported by its module, then all is well, 
since the UNCONDITIONAL bit enables the required read.

> Otherwise:
> 
>  -- If A is in a different module than PL's lookup class, then the resulting 
> lookup object has lookup mode UNCONDITIONAL. ???

Still UNCONDITIONAL + PUBLIC.  When the teleported lookup is used, it also 
ignores read edges.

> -- 

Re: MethodHandles.Lookup and modules

[John Rose]

On Dec 14, 2015, at 7:02 AM, Alan Bateman  wrote:
> ...
> In graph terms then the vertices in the readability graph are the modules. A 
> directed edge from module M to module M2 means that M reads M2. The graph is 
> mutable in that edges can be added via the API at runtime. Code in module M 
> can add a directed edge so that M reads M2. Code in M2 might add a read edge 
> in the other direction. Code in a module cannot change other modules: M 
> cannot change M2 so that M2 reads M or M3 or any other module. For 
> completeness then I should say that edges are only removed when the the 
> vertices (modules) they connect are GC'ed.

This graph of the "M1 READS M2" relation is a handy visualization.
How does it interact with exports, and the unconditional/qualified distinction?
Let me try to figure it out…

I guess the READS edges would be labeled by the types that flow across them.
Let's say TYPES_VIA(M1 READS M2) = {m2pkg.P, m2pkg.Q, …} is the set of
types (all public) from M2 that M1 can read.

And, I guess that TYPES_VIA(M1 READS M2) contains exactly
M2's unconditional (non-qualified) exports, plus M2's qualified
(non-unconditional) exports to M1.

Which is not to say that code in M1 actually uses all those types,
but it could.  And it can't use anything else from M2.

Yes?  (TIA)

>> A. Agent with full-power lookup wants to invoke another agent with the 
>> lookup,
>> but wants to limit access, because he doesn't fully trust the other agent.
>> He does a single L.in(A) to a remote-enough type A, creating a 
>> non-full-power lookup.
>> (Note:  Picking A is sometimes non-trivial.  This might be an API flaw.)
>> 
>> B. …
> 
> Case A is where our current approach might be too limited. This may be tied 
> into the discussion point as to how to choose A. If A is in the same module 
> as LC then it's as before. However if code in named module M creates a 
> full-power lookup and chooses A in another module M2 then the resulting 
> L.in(A) has zero access.
> 
> It wouldn't be hard to change this to allow PUBLIC be preserved so that 
> L.in(A) would at least allow access to public types in packages that are 
> exported unconditionally by the modules that m(A) reads. Would that increase 
> usefulness? Would such cases be cases where PL is equally useful?

Hmm…  What PL does is ignore the M1 READS M2 graph, or (equivalently)
adds temporary edges as needed to access unconditionally exported names.

(Anticipating a reply to Alex's message of today…)  Possibly, a PUBLIC-type
lookup QL would only be able to see types readable by M1=MODULE(QL.LC).
That is, QL (possibly) would respect the READS graph as it exists, and not
attempt to extend it on the fly like PL (publicLookup) does.  Points about this:

1. Such a lookup QL is stronger than PL in that it might see qualified exports
readable by M1.  (This would be Alex's QUALIFIED mode, which I like.)
But QL would also be weaker than PL, in that it would respect the existing
READS edges, and not create temporary edges like PL does.  (Am I right
that this is a real distinction?  I'm coming up to speed here!)

2. The restriction on QL is interesting in theory but maybe not in practice.
After all, if a nosy browser of modules wants to see an unconditional export
from M2, he doesn't need to bother with a QL in M1; he just uses PL.

3. How much do we all care that modules are completely leaky in their
unconditional (unqualified) exports?  It's really helpful that a module
can hide its unexported types (and seal them up for optimization, etc.).
It's also helpful that modules can use qualified exports to be friends.
Do we (can we) discourage Java apps from browsing modules
for their unconditional exports?  I'm guessing "no (and no)" is the answer.
If that's true then there is little reason to limit any QL's ability to be
a superset of PL.

4. OTOH, suppose a user is trying to be a good citizen of Module World,
and wants to avoid accessing unqualified exports in the absence of a
previously declared READS edge.  It that a real use case?  If so, then
having a QL that throws an error if a READS edge is missing will help
debug the READS graph, instead of sweeping errors under the rug.
From this point of view, usage of PL would be slightly bug-prone, since
PL ignores the carefully constructed READS graph.

Bottom line:  If the READS graph provides useful restrictions, even
for unconditional (unqualified) exports, then it would be useful to
have a QL (not stronger than PL) which emulates a lookup from
some M1, including the *inability* to read any old export..

Bottom line #2:  I like Alex's suggestion of having a QL which
mimics the lookup "perspective" of public types visible to some M1,
but external to M1.  (This QL would be not weaker than PL, as well
as not stronger.)  I was kind of fishing for that extra "mode bit",
and I think we caught something.

Bottom line #3:  If we believe that QL should be incommensurable
with PL, it follows that PL has unique 

Re: MethodHandles.Lookup and modules

[John Rose]

On Dec 17, 2015, lookup modest 12:44 PM, Alex Buckley  
wrote:
> 
> … please consider the following design:
> 
> - PUBLIC lookup mode means:
> 
>  Any 'public' type of any package exported unconditionally by the package's 
> module.

Accessing those is the uniquely special power of PL=publicLookup, which 
cavalierly
ignores the READS graph, and therefore does not need  to be computed relative 
to anything.

For clarity I would like to give this thing a special name, UNCONDITIONAL.

UNCONDITIONAL(M1) = { T, if IS_PUBLIC(T), in PACKAGE_TYPES(P), if 
IS_UNCONDITIONALLY_EXPORTED(P), for P in MODULE_PACKAGES(M1) }

UNCONDITIONAL() = UNION { UNCONDITIONAL(M1), for M1 in ALL_MODULES }

> - QUALIFIED lookup mode means:
> 
>  Any 'public' type of any package exported in qualified fashion by the 
> package's module to the lookup class's module.

QUALIFIED(M1) = { T, if IS_PUBLIC(T), for T in PACKAGE_TYPES(P), if 
IS_CONDITIONALLY_EXPORTED(P, M1), for P in MODULE_PACKAGES(M2) }

L_QUALIFIED(L) = QUALIFIED(TYPE_MODULE(L.LC))

N.B. This leaves the following related expression as an orphan (cannot be 
recomposed from the others):

- READABLE(M1) lookup mode means:

Any 'public' type of any package exported in qualified or unqualified fashion 
by the package's module to the lookup class's module.

READABLE(M1) = QUALIFIED(M1) + UNQUALIFIED(M1)
  where UNQUALIFIED(M1) = UNION { UNCONDITIONAL(M2), for M1 READS M2 in 
READS_GRAPH }

Note that UNQUALIFIED(M1) is a subset of UNCONDITIONAL() for all M1.

These classifications are nicely disjoint, so the eventual nesting
behavior arises by disjoint union.

If PUBLIC/UNCONDITIONAL is the wrong ground-level default (my point today),
then READABLE(M1) is more useful than QUALIFIED(M1).

> - MODULE lookup mode means:
> 
>  Any 'public' type of any package in the lookup class's module.

MODULE(M1) = { T, if IS_PUBLIC(T), for T in PACKAGE_TYPES(P), for P in 
MODULE_PACKAGES(M1)  }

MODULE(M1) is disjoint from READABLE(M1) and QUALIFIED(M1), but not from 
PUBLIC/UNCONDITIONAL.

For lookups we can derive things like:

L_UNCONDITIONAL(L) = UNCONDITIONAL()
L_READABLE(L) = READABLE(TYPE_MODULE(L.LC))
L_MODULE(L) = MODULE(TYPE_MODULE(L.LC))

> (Sidebar: QUALIFIED is split from MODULE primarily to be explicit about 
> access rights and secondarily to support more precise slicing of access 
> rights in a future MethodHandles.Lookup API. Example: give me a lookup object 
> to access the types in this module that offer a contract, i.e. are declared 
> 'public' without regard to exported-ness. Example: give me a lookup object to 
> access the types outside this module which are exported to it by its friends.)

This is a useful building block, but needs to be associated with 
UNQUALIFIED(M1) or UNCONDITIONAL().

> - Start with an arbitrary class in an arbitrary module calling 
> MethodHandles.Lookup.lookup() to get a "full power" lookup object L. L's 
> lookup modes are PUBLIC + QUALIFIED + MODULE + PROTECTED + PACKAGE + PRIVATE.

Or, in terms of my previous message, it could omit PUBLIC/UNCONDITIONAL and be 
READABLE + MODULE + PROTECTED + PACKAGE + PRIVATE

This means that PL is not a subset of all other non-trivial lookups.
It also means that the read-graph blindness stays unique to PL.

> - The arbitrary class obtains a Class object representing class A, then calls 
> L.in(A):
> 
>  -- If A is in a different module than L's lookup class, then the resulting 
> lookup object has lookup mode PUBLIC.

If we started with READABLE instead of PUBLIC/UNCONDITIONAL, the resulting 
lookup
L.in(A) would be degenerate (call it NOACCESS mode).  This surprise would be 
the cost
of recognizing and enforcing the uniqueness of PL.  At the present moment, I 
think
this is the best way to go.

PL.in(A) would continue to be UNCONDITIONAL (as long as A is in 
UNCONDITIONAL()).

(There's no union or non-empty intersection of L and PL, if they do 
incommensurate things.)

For PL I think this simple rule suffices:
>  -- If L is PUBLIC, then L.in(A) is also PUBLIC, unless A is inaccessible to 
> L, in which case L.in(A) has no access.

In fact, a PUBLIC/UNCONDITIONAL lookup is always the result of publicLookup()
itself or a derivative of publicLookup() via a chain of L.in(A).

BTW, there are three global rules that interact with these rules:

-- If A is not accessible to L's lookup class, using L's lookup modes, then 
L.in(A) has no access.

-- In all cases, the lookup modes of L.in(A) are a subset of (or equal to) the 
lookup modes of L.

-- In all cases (assuming no concurrent change in schemata) the set of names 
accessible to L.in(A) is a subset of (or equal to) the set of names accessible 
to L.

The third rule is a very broad requirement with all sorts of detailed 
implications.
All the other rules are designed to add up to the this main rule.

>  -- If A is in the same module as L's lookup class, but a different package, 
> then the resulting lookup object has lookup mode PUBLIC + 

Re: MethodHandles.Lookup and modules

[John Rose]

On Dec 10, 2015, at 11:20 AM, stanislav lukyanov 
 wrote:
> 
> Alan,
> thanks for the comments!
> Please see below.
> 
> On 10.12.2015 1:16, Alan Bateman wrote:
>> 
>> On 09/12/2015 14:54, stanislav lukyanov wrote:
>>> 
>>> 1) Lookup.in() javadoc says
>>> "If the lookup for this Lookup is in a named module, and the new lookup 
>>> class is in a different
>>> module M, then no members, not even public members in M's exported 
>>> packages, will be accessible"
>>> However, it seems to be possible to have PUBLIC access after
>>> lookupInA.in(ClassInB.class)
>>> without breaking "no more access capabilities than the original" rule.
>>> 
>>> Are such transitions intentionally forbidden?
>> Yes, this is intention as the lookup class and mode bits would otherwise not 
>> be sufficient when you teleport through a sequence of lookup classes. It 
>> might be clearer if you extend the example to 
>> in(B.class).in(C.class).in(D.class) where B, C and D are in different named 
>> modules and work out the intersection.
> This is connected to the second part of the question (2).
> If access to readable modules were allowed only if MODULE bit is set,
> transition from A to B would be safe, since only one transition could be made 
> (actually, same as it is now, but not from unnamed module only).
> BTW, 'requires public' chain could be traversed safely even with PUBLIC mode:
> A.in(B).in(C).in(D) would have access to D's exported members and D's 
> 'requires public' - same as A.
> 
> WDYT?

This stuff makes my head hurt, but I'm fine with any semantics that preserves 
the following:

1. full power: MethodHandles.lookup() has the same privileges as 
(non-constructor) bytecode in the caller class

2. downward monotonic
2a. L.in(A) never has more privileges than L (for all L, A)
2b. L.in(A) never has more privileges than a full power lookup on A (for all L, 
A)
2c. as a corollary, a chain L.in(A).in(B).in(C)… has no more privileges than L 
or any lookups in A, B, C, …

3. graceful degradation:  L.in(A) loses only privileges broken by the distance 
between LC=L.lookupClass and A
3a. if A==LC no privileges are lost; the identical L can be the result
3b. if A and LC are nestmates, only protected privileges may be lost (dynamic 
emulation of JLS nestmate access)
3c. if A and LC are in the same package, only private privileges may be lost
3d. if A and LC are in the same module, only package privileges may be lost

4. downward convergence (to publicLookup or empty privileges)
4a. if A is inaccessible to LC=L.lookupClass, L.in(A) has no privileges (less 
than publicLookup)
4b. if A is accessible to LC and L has non-empty privileges, L.in(A) is no less 
privileged than publicLookup
4c. for any L with non-empty privileges, there is a sequence of types A,B where 
L.in(A).in(B) is equivalent to publicLookup

5. publicLookup has a reasonable minimal set of globally acceptable privileges
5a. this set of privileges is singular, and does not depend on the lookupClass
5b. the only possible results of publicLookup.in(A) are no change, and falling 
to empty privileges

Access can only be measured against the current state of the module graph,
which means that certain results can vary over time, in a consistent way
across the whole system.

The above are my preferences; I can imaging tweaking some of those things
in order to make the API fit more gracefully into modules.  I rely on Alan & co.
to figure that out!


>>> 2) Lookup.in() javadoc says
>>> "If the lookup class for this Lookup is not in a named module, and the new 
>>> lookup class is
>>> in a named module M, then no members in M's non-exported packages will be 
>>> accessible"
>>> 
>>> Spec says nothing about M's 'requires' and 'exports to M'.
>>> In current implementation, modules required by M are still accessible
>>> and packages exported to M (with 'exports to')  are not accessible.
>>> 
>>> It seems right that packages exported to M are not accessible.
>>> Should it be specified in the javadoc, or is it implied due to definitions 
>>> somewhere else?
>> It is implied because the javadoc provides the guarantee that the resulting 
>> Lookup has no more access than the original. In this case there are packages 
>> exported by M's friends to M that are not accessible to code in unnamed 
>> modules. However, I think you are right that this could be make clearer in 
>> the javadoc. I'm sure that once the design settles down that we'll do 
>> several passes over the javadoc.
> OK, thanks!
> Is there a JBS RFE that aggregates such issues, or anything like that?

(BTW, see https://bugs.openjdk.java.net/browse/JDK-8145070 )

>> 
>>> :
>>> 
>>> However, this creates some uncertainty.
>>> Suppose we have A requires B. From user's point resulting lookup of
>>> publicLookup().in(ClassInA.class)
>>> may or may not have access to the module B - it depends on module A's 
>>> 'requires' which user doesn't know about.
>>> 
>>> It may come to the following situation. 

Re: CFV: New Jigsaw Committer: Harold Seigel

[John Rose]

Vote: yes

> On Sep 11, 2015, at 12:31 PM, Karen Kinnear  wrote:
> 
> I hereby nominate Harold Seigel to Jigsaw Committer.

Re: CFV: New jigsaw Committer: Jean-Francois Denise

[John Rose]

Vote: yes

> On Sep 16, 2015, at 12:17 PM, mark.reinh...@oracle.com wrote:
> 
> Vote: yes
> 
> - Mark

Re: CFV: New Committer: Christian Tornqvist

[John Rose]

Vote: yes

> On Sep 16, 2015, at 2:42 PM, Karen Kinnear  wrote:
> 
> I hereby nominate Christian Tornqvist to Jigsaw Committer.

Re: need advice on module structure for ISA-specific sources and classes

[John Rose]

On Sep 4, 2014, at 3:08 PM, Alex Buckley alex.buck...@oracle.com wrote:

 - I see there are multiple versions of the VIS extension for SPARC, and of 
 the AVX extension for x86. Do you envisage encoding ISA versions into the 
 $CPU value? What taxonomy of ISAs is suggested by JNR?

The current set of raw processor names in libffi is:
  arm i386 mips mipsel ppc s390 s390x sparc sparcv9 x86_64

In HotSpot sources, we find it helpful to organize the x86 and sparc variants 
together, which may or may not be the right answer here.  There might be 
similar arguments for mips, s390, and arm variants.

The x86 and sparc sources can go together, even though the old 32-bit and new 
64-bit ISAs are very different.  That is because there are enough similarities 
to justify some common code, and also because the chip vendors usually give 
enough hooks (cpu version queries) to tell the versions apart dynamically.  
I.e., once you know the processor family, you can often work from a common 
source base.

Dunno whether that theory will work for arm64.

 - If precision is desirable, can I suggest preferring isa/$ISA to cpu/$CPU 
 throughout? First, because we don't want to sling CPU-ish terms like x86 
 when we mean ISA-ish terms like x64-with-AVX2. Second, because I think 
 Java's heritage is more ISA than CPU (see JVMS section 1.2).

I confess to being confused between CPU and ISA, and also between uname -m 
-i -p etc.  I think you are right that ISA is a more correct term than CPU; 
by saying CPU I'm mainly following the HotSpot convention which might be an 
outlier.

If we are going for precision, ISA (although more inclusive than CPU in the 
strict sense) may be too precise since it distinguishes several layers of 
x86_64 (to say nothing of the 32- vs. 64-bit operating modes).

 - With reference to mlib_v_ImageCopy_blk.s: it is OS-specific, 
 SPARCv9-with-VIS-specific code, right? So its path would be something like:
 
 src/java.desktop/unix.sparc9vis/native/libawt/sun/...

Probably just unix.sparc (under the above theory about keeping twins together) 
or unix.sparcv9.  There are several layers of vis, but they can be 
distinguished down in the weeds using cpu version sensing, ifdefs, etc.

Thanks for helping air this out.

— John