Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[mark . reinhold]

(Finally getting back to this thread, now that vacation season is over.)

2016/7/21 10:01:11 -0700, jason.gre...@redhat.com:
> On Jul 18, 2016, at 4:30 PM, mark.reinh...@oracle.com wrote:
>> 2016/7/13 20:27:45 -0700, jason.gre...@redhat.com:
>>> ...
>>> 
>>> Sorry for the confusion, what I was trying to say on this point was a bit
>>> different. What I was trying to say was:
>>> 
>>> (2') It weakens encapsulation by forcing the introduction of exports
>>>   introducing potential conflicts that break applications.
>>> 
>>> As an example, assume I have three modules with classloader-per-module
>>> isolation (A, B, and Victim)
>>> 
>>> - A exports foo, and has a non-exported package “bar"
>>> 
>>> - B exports bar
>>> 
>>> - Victim has a module-info with requires A; requires B
>>> 
>>> Now A decides to use IoC on some of its classes in bar, so it’s
>>> definition is changed to:
>>> 
>>> { exports foo; exports dynamic bar; }
>>> 
>>> Since exports dynamic is internally a normal export at runtime, module
>>> resolution fails when loading Victim, because its now including a
>>> duplicate package, even though A had no intention of publishing its
>>> internal bar package for linkage.
>> 
>> Got it.  Thanks for clarifying this -- I agree that it's a problem.
>> 
>> Fortunately I think we can address it simply by revising the semantics of
>> `exports dynamic p` to omit the package-conflict constraint.  This would
>> allow split packages to occur more readily at run time, though still
>> really only in fairly obscure situations involving poorly-written class
>> loaders.
> 
> That would help, but there is also class visibility issues that would
> need to be addressed as well.
> 
> Example 1 (Ambiguous class names):
> 
> Both A and B export “bar”, and both define “bar.MyClass” which have
> differing definitions. Victim could load the supposed to be hidden A’s
> MyClass instead of the intended B’s MyClass.
> 
> There is also a variant of this where the conflict is between Victim
> and A if A also exports another hidden package that is present in
> Victim itself.

Alan addressed this in a nearby message.

A high-level point worth emphasizing here is that visibility issues are
class-loader issues, and Jigsaw (for the most part) does not dictate how
custom class loaders should work.  It's a good idea for class loaders to
respect the readability relationships set up by the module system, but if
they don't then there's nothing that the module system can really do
about it.

> Example 2 (Unintentional discovery):
> 
> Victim uses ClassLoader.getResources (plural), looking for a standard
> configuration file or class name, and receives entries for both A and
> B. A’s was not intended to be discovered by victim, and leads to a
> failure state. As an example perhaps the configuration file in B
> specifies a class name in B’s dependency, which is not visible to
> Victim. Or, perhaps A’s config leads to duplicate runtime actions being
> configured (since the file was really only indented for A, which also
> processes it)

... which you later amended:

> Sorry for the confusion, this should read:
> 
> " As an example perhaps the configuration file in [A] specifies a class
> name in [A]’s dependency, which is not visible to Victim. Or, perhaps A’s
> config leads to duplicate runtime actions being configured (since the
> file was really only indented for A, which also processes it)"

Alan also addressed this, in a couple of different nearby messages.
Again, what the class loaders do in this example is not really up to the
module system.  This example can, however, be seen as an argument in
favor of #ResourceEncapsulation, as Alan noted.  If module A contains a
resource that's strictly internal to A, and if the module system gives
the author of A a way to encapsulate such resources, then the module
system could help out in this case by refusing to locate that resource.
(Resource location is one way in which Jigsaw may well change how all
class loaders work.)

> You can potentially address 1 with precedence, but not 2. 
> 
> I think you would need to say that export dynamic is only utilizable
> for reflection permissions and has no other similarity with “export”
> (although perhaps that’s what you meant?)

No, that's not what I meant.  `exports dynamic` allows static (i.e.,
bytecode) references too; it just doesn't allow compile-time access.
Anyway, this isn't an accessibility issue, as noted.

> If you combine that approach with a wildcard capability like you
> mentioned earlier then I’ll admit its very hard for me to quibble over
> a one line additional requirement in module-info.java.

Glad to hear it.

> Although, for completeness, let me (re?)introduce one other
> consideration that was briefly mentioned (although with sparing
> details) earlier in the thread
> 
> If you have a custom serialization framework that is supposed to be
> identical to Java serialization in contract, then it becomes impossible
> to mirror using the only 

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Alan Bateman]

On 23/07/2016 01:32, Jason Greene wrote:


:
This was intended to mirror the same structure as the original example as well: 
A exports dynamic bar, and B exports bar, and also all modules are following a 
class loader per module paradigm.  I had a follow-up that corrected this, it 
should have read:

"As an example perhaps the configuration file in [A] specifies a class name in 
[A]’s dependency, which is not visible to Victim. Or, perhaps A’s config leads to 
duplicate runtime actions being configured (since the file was really only indented 
for A, which also processes it)”

It sounds from your description above that Victim’s classloader would not 
delegate to A’s class loader, so resources in A’s /bar would not be 
discoverable from Victim. Extending this scenario as you describe with B 
requires C.  I imagine that C’s contents would not be visible to Victim (unless 
it was defined B requires public C)?

I'm sure Mark will continue this thread but just to say that there is 
another open issue (#ResourceEncapsulation) on the JSR issues list that 
may be where you are going. If there is a resource in A (for use by A) 
then it may or may be not be located by code in other modules, that is 
part of the other discussion.


As regards the specific scenario, where a container arranges for a class 
loader per module, then Victim's class loader should not be delegating 
to A's loader, at least not for bar.* types and in the context of the 
updated semantics suggested earlier in the thread. This is of course not 
something that the module system really controls as you can map modules 
to class loaders and do delegation as you want (the module system places 
very few restrictions). The legacy getResource* methods are not 
specified to work with graphs of class loaders and not possible to say 
what the overridden implementations might do. So I think all we can 
really say here is that if the name of a type in C leaks to Victim then 
it may or may not be able to get a Class reference. It's no different to 
JDK 8 in this regard. It is of course possible that a C type gets leaked 
to Victim by other means. In that case then only the public types in C's 
exported packages will be accessible to Victim.


-Alan

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Peter Levart]

Hi,

Just one thougt...

It seems that visibility of resources should be governed by some module
policy. It is becoming apparent that global visibility of resources where
only class loader delegation matters can sometimes be a problem. OTOH,
totally encapsulating all resources is also not acceptable. It would be
nice if module descriptor could specify which resources are "exported" and
which visible only to the owner module.

Regards, Peter

On Jul 23, 2016 2:33 AM, "Jason Greene"  wrote:

>
> > On Jul 21, 2016, at 3:45 PM, Alan Bateman 
> wrote:
> >
> > Just a few comments on the examples posed in the last mail.
>
> Thanks!
>
> >
> >
> > On 21/07/2016 18:01, Jason Greene wrote:
> >> :
> >> That would help, but there is also class visibility issues that would
> need to be addressed as well.
> >>
> >> Example 1 (Ambiguous class names):
> >>
> >> Both A and B export “bar”, and both define “bar.MyClass” which have
> differing definitions. Victim could load the supposed to be hidden A’s
> MyClass instead of the intended B’s MyClass.
> >>
> >> There is also a variant of this where the conflict is between Victim
> and A if A also exports another hidden package that is present in Victim
> itself.
> > If the scenario were:
> >
> > A exports bar
> > B exports bar
> > Victim requires A; requires B;
> >
> > then it would be a split package issue and would be rejected at
> resolution time (or more specifically, one of the post resolution checks).
> The simple rule is that two or more modules can't export the same package
> to a module that reads both. This includes the case where a module M
> containing package p reads another module that exports p to M.
> >
> > If the scenario is:
> >
> > A exports dynamic bar
> > B exports bar
> > Victim requires A; requires B;
>
> Sorry, I should have been more clear that I was reusing the earlier
> example that was immediately before the reply. This latter scenario you
> describe is what I intended.
>
> >
> > then, with the updated semantics, there is no split package issue at
> resolution time. Victim reads module A and module B but the `exports
> dynamic` edges are ignored by the post-resolution checks. If it "as if"
> Victim invokes addReads(A), nothing more
> >
> > Moving on to the class loader delegation graph then it needs to support
> the readability graph and exports. With the updated semantics then any
> reference to bar.MyClass in Victim will delegate to the loader of module B.
> Victim's loader doesn't know anything about package bar in module A.
> >
> > There are of course dozens of ways in which a Class object for some type
> bar.* type in module A could leak to Victim, maybe it uses Class.forName
> specifying module A's loader. Assuming code in Victim gets a reference to a
> public type in module A's bar package then it will be accessible to code in
> Victim (as intended in the scenario I think).
>
> Ok great, that’s the behavior I was advocating. My original intention was
> more that a separate unrelated module (e.g. DI framework used by A) would
> dynamically access A, and that victim was just the unintended consequence
> of A enabling the unrelated module. However, the behavior of Victim being
> able to access it as well makes sense and is completely consistent with the
> definitions above and past behavior.
>
> >
> >
> >> Example 2 (Unintentional discovery):
> >>
> >> Victim uses ClassLoader.getResources (plural), looking for a standard
> configuration file or class name, and receives entries for both A and B.
> A’s was not intended to be discovered by victim, and leads to a failure
> state. As an example perhaps the configuration file in B specifies a class
> name in B’s dependency, which is not visible to Victim. Or, perhaps A’s
> config leads to duplicate runtime actions being configured (since the file
> was really only indented for A, which also processes it)
> > In this scenario then I can't tell if the "should-not-be-discovered" A
> type is an exported package or not. If it's a public type in a package
> exported to Victim then it will be accessible to code in Victim. If it's
> not in an exported package then it won't be accessible.
> >
> > Whether it's visible is another question, that depends on how the class
> loaders and delegation are arranged. In the simple case then A, B, and
> Victim are all defined to the same class loader. That would allow Victim to
> load any type in A but only the public types in A's exported packages would
> be  accessible.
> >
> > For the scenario where the file lists some class in a random module B
> reads, say module C, then it may or may not be visible to Victim. It might
> be able to load it, it might not. Assuming it is visible then it may or may
> not be accessible. There isn't enough in the example scenario to know if
> this type is in a packaged exported by C or not.
>
> This was intended to mirror the same structure as the original example as
> well: A exports 

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Jason Greene]

> On Jul 21, 2016, at 3:45 PM, Alan Bateman  wrote:
> 
> Just a few comments on the examples posed in the last mail.

Thanks!

> 
> 
> On 21/07/2016 18:01, Jason Greene wrote:
>> :
>> That would help, but there is also class visibility issues that would need 
>> to be addressed as well.
>> 
>> Example 1 (Ambiguous class names):
>> 
>> Both A and B export “bar”, and both define “bar.MyClass” which have 
>> differing definitions. Victim could load the supposed to be hidden A’s 
>> MyClass instead of the intended B’s MyClass.
>> 
>> There is also a variant of this where the conflict is between Victim and A 
>> if A also exports another hidden package that is present in Victim itself.
> If the scenario were:
> 
> A exports bar
> B exports bar
> Victim requires A; requires B;
> 
> then it would be a split package issue and would be rejected at resolution 
> time (or more specifically, one of the post resolution checks). The simple 
> rule is that two or more modules can't export the same package to a module 
> that reads both. This includes the case where a module M containing package p 
> reads another module that exports p to M.
> 
> If the scenario is:
> 
> A exports dynamic bar
> B exports bar
> Victim requires A; requires B;

Sorry, I should have been more clear that I was reusing the earlier example 
that was immediately before the reply. This latter scenario you describe is 
what I intended. 

> 
> then, with the updated semantics, there is no split package issue at 
> resolution time. Victim reads module A and module B but the `exports dynamic` 
> edges are ignored by the post-resolution checks. If it "as if" Victim invokes 
> addReads(A), nothing more
> 
> Moving on to the class loader delegation graph then it needs to support the 
> readability graph and exports. With the updated semantics then any reference 
> to bar.MyClass in Victim will delegate to the loader of module B. Victim's 
> loader doesn't know anything about package bar in module A.
> 
> There are of course dozens of ways in which a Class object for some type 
> bar.* type in module A could leak to Victim, maybe it uses Class.forName 
> specifying module A's loader. Assuming code in Victim gets a reference to a 
> public type in module A's bar package then it will be accessible to code in 
> Victim (as intended in the scenario I think).

Ok great, that’s the behavior I was advocating. My original intention was more 
that a separate unrelated module (e.g. DI framework used by A) would 
dynamically access A, and that victim was just the unintended consequence of A 
enabling the unrelated module. However, the behavior of Victim being able to 
access it as well makes sense and is completely consistent with the definitions 
above and past behavior.

> 
> 
>> Example 2 (Unintentional discovery):
>> 
>> Victim uses ClassLoader.getResources (plural), looking for a standard 
>> configuration file or class name, and receives entries for both A and B. A’s 
>> was not intended to be discovered by victim, and leads to a failure state. 
>> As an example perhaps the configuration file in B specifies a class name in 
>> B’s dependency, which is not visible to Victim. Or, perhaps A’s config leads 
>> to duplicate runtime actions being configured (since the file was really 
>> only indented for A, which also processes it)
> In this scenario then I can't tell if the "should-not-be-discovered" A type 
> is an exported package or not. If it's a public type in a package exported to 
> Victim then it will be accessible to code in Victim. If it's not in an 
> exported package then it won't be accessible.
> 
> Whether it's visible is another question, that depends on how the class 
> loaders and delegation are arranged. In the simple case then A, B, and Victim 
> are all defined to the same class loader. That would allow Victim to load any 
> type in A but only the public types in A's exported packages would be  
> accessible.
> 
> For the scenario where the file lists some class in a random module B reads, 
> say module C, then it may or may not be visible to Victim. It might be able 
> to load it, it might not. Assuming it is visible then it may or may not be 
> accessible. There isn't enough in the example scenario to know if this type 
> is in a packaged exported by C or not.

This was intended to mirror the same structure as the original example as well: 
A exports dynamic bar, and B exports bar, and also all modules are following a 
class loader per module paradigm.  I had a follow-up that corrected this, it 
should have read:

"As an example perhaps the configuration file in [A] specifies a class name in 
[A]’s dependency, which is not visible to Victim. Or, perhaps A’s config leads 
to duplicate runtime actions being configured (since the file was really only 
indented for A, which also processes it)”

It sounds from your description above that Victim’s classloader would not 
delegate to A’s class loader, so resources in A’s 

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Alan Bateman]

Just a few comments on the examples posed in the last mail.


On 21/07/2016 18:01, Jason Greene wrote:

:
That would help, but there is also class visibility issues that would need to 
be addressed as well.

Example 1 (Ambiguous class names):

Both A and B export “bar”, and both define “bar.MyClass” which have differing 
definitions. Victim could load the supposed to be hidden A’s MyClass instead of 
the intended B’s MyClass.

There is also a variant of this where the conflict is between Victim and A if A 
also exports another hidden package that is present in Victim itself.

If the scenario were:

A exports bar
B exports bar
Victim requires A; requires B;

then it would be a split package issue and would be rejected at 
resolution time (or more specifically, one of the post resolution 
checks). The simple rule is that two or more modules can't export the 
same package to a module that reads both. This includes the case where a 
module M containing package p reads another module that exports p to M.


If the scenario is:

A exports dynamic bar
B exports bar
Victim requires A; requires B;

then, with the updated semantics, there is no split package issue at 
resolution time. Victim reads module A and module B but the `exports 
dynamic` edges are ignored by the post-resolution checks. If it "as if" 
Victim invokes addReads(A), nothing more.


Moving on to the class loader delegation graph then it needs to support 
the readability graph and exports. With the updated semantics then any 
reference to bar.MyClass in Victim will delegate to the loader of module 
B. Victim's loader doesn't know anything about package bar in module A.


There are of course dozens of ways in which a Class object for some type 
bar.* type in module A could leak to Victim, maybe it uses Class.forName 
specifying module A's loader. Assuming code in Victim gets a reference 
to a public type in module A's bar package then it will be accessible to 
code in Victim (as intended in the scenario I think).




Example 2 (Unintentional discovery):

Victim uses ClassLoader.getResources (plural), looking for a standard 
configuration file or class name, and receives entries for both A and B. A’s 
was not intended to be discovered by victim, and leads to a failure state. As 
an example perhaps the configuration file in B specifies a class name in B’s 
dependency, which is not visible to Victim. Or, perhaps A’s config leads to 
duplicate runtime actions being configured (since the file was really only 
indented for A, which also processes it)
In this scenario then I can't tell if the "should-not-be-discovered" A 
type is an exported package or not. If it's a public type in a package 
exported to Victim then it will be accessible to code in Victim. If it's 
not in an exported package then it won't be accessible.


Whether it's visible is another question, that depends on how the class 
loaders and delegation are arranged. In the simple case then A, B, and 
Victim are all defined to the same class loader. That would allow Victim 
to load any type in A but only the public types in A's exported packages 
would be  accessible.


For the scenario where the file lists some class in a random module B 
reads, say module C, then it may or may not be visible to Victim. It 
might be able to load it, it might not. Assuming it is visible then it 
may or may not be accessible. There isn't enough in the example scenario 
to know if this type is in a packaged exported by C or not.


-Alan.

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Jason Greene]

> On Jul 21, 2016, at 12:01 PM, Jason Greene  wrote:
> 
> 
>> On Jul 18, 2016, at 4:30 PM, mark.reinh...@oracle.com wrote:
>> 
> 
> Hi Mark, Thanks for the reply. I have snipped out portions to make it easier 
> to follow the thread.
> 
>>> 
>>> I agree they are certainly intermixed elements of a system, but I’d also
>>> argue IoC is pervasive in SE applications as well (e.g. inclusion of 330
>>> and 250 in SE are examples of a desire for SE usage). I can’t refute that
>>> it has greater usage in EE, since its part of the spec, and thus
>>> effectively every EE application.
>> 
>> FYI, JSR 330 (DI annotations) is not in Java SE, though it's certainly
>> used in Java SE applications in combination with various DI frameworks.
>> 
>> JSR 250 ("common" annotations) specifies 14 annotations, but just five
>> of them are in Java SE.  They're really only there to support JAX-WS, a
>> component shared with Java EE.  So far as I know they're not used much
>> in SE applications except in conjunction with JAX-WS.
> 
> Ah yes, of course thanks for the correction. Not sure why I had it in my head 
> that 330 was included.
> 
> -snip-
> 
> 
>>> 
>>> Sorry for the confusion, what I was trying to say on this point was a bit
>>> different. What I was trying to say was:
>>> 
>>> (2') It weakens encapsulation by forcing the introduction of exports
>>>   introducing potential conflicts that break applications.
>>> 
>>> As an example, assume I have three modules with classloader-per-module
>>> isolation (A, B, and Victim)
>>> 
>>> - A exports foo, and has a non-exported package “bar"
>>> 
>>> - B exports bar
>>> 
>>> - Victim has a module-info with requires A; requires B
>>> 
>>> Now A decides to use IoC on some of its classes in bar, so it’s
>>> definition is changed to:
>>> 
>>> { exports foo; exports dynamic bar; }
>>> 
>>> Since exports dynamic is internally a normal export at runtime, module
>>> resolution fails when loading Victim, because its now including a
>>> duplicate package, even though A had no intention of publishing its
>>> internal bar package for linkage.
>> 
>> Got it.  Thanks for clarifying this -- I agree that it's a problem.
>> 
>> Fortunately I think we can address it simply by revising the semantics of
>> `exports dynamic p` to omit the package-conflict constraint.  This would
>> allow split packages to occur more readily at run time, though still
>> really only in fairly obscure situations involving poorly-written class
>> loaders.

-snip -

> Example 2 (Unintentional discovery):
> 
> Victim uses ClassLoader.getResources (plural), looking for a standard 
> configuration file or class name, and receives entries for both A and B. A’s 
> was not intended to be discovered by victim, and leads to a failure state. As 
> an example perhaps the configuration file in B specifies a class name in B’s 
> dependency, which is not visible to Victim. Or, perhaps A’s config leads to 
> duplicate runtime actions being configured (since the file was really only 
> indented for A, which also processes it)

Sorry for the confusion, this should read:

" As an example perhaps the configuration file in [A] specifies a class name in 
[A]’s dependency, which is not visible to Victim. Or, perhaps A’s config leads 
to duplicate runtime actions being configured (since the file was really only 
indented for A, which also processes it)”

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Jason Greene]

> On Jul 18, 2016, at 4:30 PM, mark.reinh...@oracle.com wrote:
> 

Hi Mark, Thanks for the reply. I have snipped out portions to make it easier to 
follow the thread.

>> 
>> I agree they are certainly intermixed elements of a system, but I’d also
>> argue IoC is pervasive in SE applications as well (e.g. inclusion of 330
>> and 250 in SE are examples of a desire for SE usage). I can’t refute that
>> it has greater usage in EE, since its part of the spec, and thus
>> effectively every EE application.
> 
> FYI, JSR 330 (DI annotations) is not in Java SE, though it's certainly
> used in Java SE applications in combination with various DI frameworks.
> 
> JSR 250 ("common" annotations) specifies 14 annotations, but just five
> of them are in Java SE.  They're really only there to support JAX-WS, a
> component shared with Java EE.  So far as I know they're not used much
> in SE applications except in conjunction with JAX-WS.

Ah yes, of course thanks for the correction. Not sure why I had it in my head 
that 330 was included.

-snip-


>> 
>> Sorry for the confusion, what I was trying to say on this point was a bit
>> different. What I was trying to say was:
>> 
>> (2') It weakens encapsulation by forcing the introduction of exports
>>introducing potential conflicts that break applications.
>> 
>> As an example, assume I have three modules with classloader-per-module
>> isolation (A, B, and Victim)
>> 
>> - A exports foo, and has a non-exported package “bar"
>> 
>> - B exports bar
>> 
>> - Victim has a module-info with requires A; requires B
>> 
>> Now A decides to use IoC on some of its classes in bar, so it’s
>> definition is changed to:
>> 
>> { exports foo; exports dynamic bar; }
>> 
>> Since exports dynamic is internally a normal export at runtime, module
>> resolution fails when loading Victim, because its now including a
>> duplicate package, even though A had no intention of publishing its
>> internal bar package for linkage.
> 
> Got it.  Thanks for clarifying this -- I agree that it's a problem.
> 
> Fortunately I think we can address it simply by revising the semantics of
> `exports dynamic p` to omit the package-conflict constraint.  This would
> allow split packages to occur more readily at run time, though still
> really only in fairly obscure situations involving poorly-written class
> loaders.

That would help, but there is also class visibility issues that would need to 
be addressed as well. 

Example 1 (Ambiguous class names):

Both A and B export “bar”, and both define “bar.MyClass” which have differing 
definitions. Victim could load the supposed to be hidden A’s MyClass instead of 
the intended B’s MyClass.

There is also a variant of this where the conflict is between Victim and A if A 
also exports another hidden package that is present in Victim itself.

Example 2 (Unintentional discovery):

Victim uses ClassLoader.getResources (plural), looking for a standard 
configuration file or class name, and receives entries for both A and B. A’s 
was not intended to be discovered by victim, and leads to a failure state. As 
an example perhaps the configuration file in B specifies a class name in B’s 
dependency, which is not visible to Victim. Or, perhaps A’s config leads to 
duplicate runtime actions being configured (since the file was really only 
indented for A, which also processes it)

You can potentially address 1 with precedence, but not 2. 

I think you would need to say that export dynamic is only utilizable for 
reflection permissions and has no other similarity with “export” (although 
perhaps that’s what you meant?)

If you combine that approach with a wildcard capability like you mentioned 
earlier then I’ll admit its very hard for me to quibble over a one line 
additional requirement in module-info.java.

Although, for completeness, let me (re?)introduce one other consideration that 
was briefly mentioned (although with sparing details) earlier in the thread

If you have a custom serialization framework that is supposed to be identical 
to Java serialization in contract, then it becomes impossible to mirror using 
the only available standard means (core reflection), since that mechanism 
disallows non-exported packages. Currently a custom serialization framework 
only needs to handle one non-standard case (missing no-arg constructor). Going 
forward it would need to use Unsafe for everything. 
> 
>> Qualified exports could in theory address this problem, but they are
>> problematic in a dynamic environment since the module is simply not in a
>> position to know all of the various modules which would/could enhance
>> it. ...
> 
> I completely agree.  In general I think it's inappropriate for the author
> of a module to write qualified exports except in some very special cases,
> and this is most definitely not one of them.
> 
>>> These observations lead to your suggestion to allow declarative module
>>> boundaries to be overridden by "trusted" framework code.  It's far from

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[mark . reinhold]

2016/7/13 16:15:33 -0700, peter.lev...@gmail.com:
> On 07/13/2016 11:47 PM, mark.reinh...@oracle.com wrote:
>> ...
>> 
>> If the container is set up to provide, e.g., Hibernate to this particular
>> application, then it could narrow the accessibility of the entity classes
>> by rewriting the above module declaration to refine the `exports dynamic`
>> directive:
>> 
>> module com.foo.data {
>> requires java.persistence;
>> exports dynamic com.foo.data.model
>>  to hibernate.core, hibernate.entitymanager;
>> }
>> 
>> (This is one of the very few use cases for qualified dynamic exports.)
>> 
>> Whether standalone or in a container the same set of packages is exported
>> by the module; the only difference is that, inside the container, the
>> exports are qualified.
> 
> What additional metadata does container need to rewrite a module 
> descriptor like in above example? Does it need the whole set of exports 
> that replace existing dynamic exports? Or only the target modules that 
> it "attaches" to all unqualified dynamic exports? If the later, then 
> what does container do if one wants to rewrite only a selection of 
> unqualified dynamic exports to target one set of modules (for example an 
> IoC implementation) and  another (possibly overlapping) selection of 
> exports to target some other set of modules (for example a JPA 
> implementation)? Does it give up and "attaches" all targets to all 
> unqualified dynamic exports ?
> 
> I think something is missing here. A kind of hook to identify or "tag" a 
> set of unqualified dynamic exports so that it can be located by the 
> container.

I think we can address this with the tools that we already have, along
the lines of what Stephane Epardaud suggests nearby.

The container can already see that the `com.foo.data.model` package
contains elements marked with annotations from the `java.persistence`
module.  If the Hibernate modules are annotated

@RefineDynamicExports({ javax.persistence.Converter.class,
javax.persistence.Entity.class,
javax.persistence.Embeddable.class,
javax.persistence.MappedSuperClass.class })
module hibernate.core { ... }

@RefineDynamicExports({ javax.persistence.Converter.class,
javax.persistence.Entity.class,
javax.persistence.Embeddable.class,
javax.persistence.MappedSuperClass.class })
module hibernate.entitymanager { ... }

then the container could refine the dynamic export of a package that
contains elements annotated with @Converter, @Entity, etc., from being
unqualified, as it was originally written, to being qualified to these
two Hibernate modules.  (It could refine unqualified dynamic wildcard
exports in the same fashion.)

In a @RefineDynamicExports annotation you wouldn't have to list all the
annotations in the relevant package but just the main ones, any one of
which whose presence implies use of the framework.

- Mark

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[mark . reinhold]

2016/7/13 20:27:45 -0700, jason.gre...@redhat.com:
> Thanks for you reply! My thoughts are inline. I apologize in advance for
> the length/verbosity. Also, as a general disclaimer, I realize that you
> are all experts; in many of my arguments, I occaisionally restate certain
> concepts that I know you are all intimately aware of to frame the
> argument. Corrections are, as always, welcome.

No worries -- we'll try, as always, to be gentle!

> On Jul 13, 2016, at 4:47 PM, mark.reinh...@oracle.com wrote:
>> To put what Alex wrote in a somewhat different way, I'd say that the
>> tension here is between explicit configuration (as one finds today in,
>> e.g., the Maven world) and implicit configuration (IoC).
> 
> Just a small nit: IoC can also be explicit, its just that the
> explicitness is decoupled from the module, and controlled by another
> party, allowing for more flexibility in the assembled system.

Sure -- I was just trying to characterize the situation from the
standpoint of the module developer rather than that of the assembler
or deployer or container implementor (or any other role).

>> Both approaches
>> are important.  The former is typical of standalone Java SE applications
>> while the latter is typical of Java EE applications, though the two
>> approaches are often intermixed.
> 
> I agree they are certainly intermixed elements of a system, but I’d also
> argue IoC is pervasive in SE applications as well (e.g. inclusion of 330
> and 250 in SE are examples of a desire for SE usage). I can’t refute that
> it has greater usage in EE, since its part of the spec, and thus
> effectively every EE application.

FYI, JSR 330 (DI annotations) is not in Java SE, though it's certainly
used in Java SE applications in combination with various DI frameworks.

JSR 250 ("common" annotations) specifies 14 annotations, but just five
of them are in Java SE.  They're really only there to support JAX-WS, a
component shared with Java EE.  So far as I know they're not used much
in SE applications except in conjunction with JAX-WS.

> I think a better use case categorization of this problem is static
> linkage vs dynamic invocation. In static linkage an explicit symbol
> mapping resolved by the language itself is ideal as it avoids ambiguity,
> and by definition is static. On the other hand with dynamic invocation
> it’s common for the caller to utilize introspection and discovery as part
> of the natural flow of executing a dynamic call. Resolving ambiguity is
> not an issue in this case, since it is already handled by the caller as
> part of introspection.

This dichotomy of implementation techniques corresponds well to the
developer-point-of-view dichotomy of explicit vs. implicit configuration
which I described earlier.

>> ...
>> 
>> If I understand correctly, your view of the present proposal is that:
>> 
>> (1) It induces too much boilerplate, requiring developers to write
>>`exports dynamic P` for every single package `P` that's subject
>>to reflection by a framework, and
> 
> That’s an accurate summary of this point. ...
> 
>> (2) It weakens encapsulation too much, by making the types in such a
>>package available for reflection at run time by any module in the
>>system.
> 
> Sorry for the confusion, what I was trying to say on this point was a bit
> different. What I was trying to say was:
> 
> (2') It weakens encapsulation by forcing the introduction of exports
>  introducing potential conflicts that break applications.
> 
> As an example, assume I have three modules with classloader-per-module
> isolation (A, B, and Victim)
> 
> - A exports foo, and has a non-exported package “bar"
> 
> - B exports bar
> 
> - Victim has a module-info with requires A; requires B
> 
> Now A decides to use IoC on some of its classes in bar, so it’s
> definition is changed to:
> 
> { exports foo; exports dynamic bar; }
> 
> Since exports dynamic is internally a normal export at runtime, module
> resolution fails when loading Victim, because its now including a
> duplicate package, even though A had no intention of publishing its
> internal bar package for linkage.

Got it.  Thanks for clarifying this -- I agree that it's a problem.

Fortunately I think we can address it simply by revising the semantics of
`exports dynamic p` to omit the package-conflict constraint.  This would
allow split packages to occur more readily at run time, though still
really only in fairly obscure situations involving poorly-written class
loaders.

> Qualified exports could in theory address this problem, but they are
> problematic in a dynamic environment since the module is simply not in a
> position to know all of the various modules which would/could enhance
> it. ...

I completely agree.  In general I think it's inappropriate for the author
of a module to write qualified exports except in some very special cases,
and this is most definitely not one of them.

>> These observations lead to your suggestion to allow 

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Stephane Epardaud]

So how about a Java Language annotation (say, @RequiresExport) that we 
could place on IoC framework annotation definitions (say, @Entity, from 
JPA) that would tell the compiler that any type annotated with @Entity 
must be exported?


That would solve the issue of making sure that users would not be 
surprised by access exceptions at run-time when Hibernate can't use 
reflection on their JPA entities. The compiler would check that, and 
IDEs would have quick-fixes to add the required (dynamic) exports to the 
module descriptor from such an error.


Now, if we want to qualify the export, it's not entirely clear to whom 
it should be exported. We could say that it has to be exported to the 
module providing the IoC annotation (JPA for @Entity), but it's likely 
not useful because it's implementations of that lib that would use 
reflection (Hibernate). If the module system was extended to provide 
"virtual" modules (for example, Hibernate would be marked as "providing" 
JPA), then we could treat Hibernate as inheriting the "requires dynamic" 
permissions of the modules it provides (JPA) and so it would be able to 
reflect over types exported to JPA.


Virtual modules is one of the most wanted features in Ceylon modularity, 
but we haven't implemented it yet, for lack of time resolving some of 
the issues it raises. Though I don't think virtual modules are required 
for this feature to be useful.


I don't think it makes much sense to add Hibernate or other JPA 
implementations as "friend modules" to the JPA module, as that would 
prevent other implementations, but perhaps an annotation (say, 
@InheritDynamicExports) on the Hibernate module (or rather, on its 
import of the JPA module) would mark it as inheriting dynamic export 
permissions from the JPA module? This way the VM would let Hibernate 
reflect over my model types even though they're only exported 
dynamically to JPA. A SecurityManager would be allowed to restrict that, 
of course.

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Jason Greene]

> On Jul 13, 2016, at 4:47 PM, mark.reinh...@oracle.com wrote:
> 
> Jason -- thanks for your feedback on this topic.

Hi Mark,

Thanks for you reply! My thoughts are inline. I apologize in advance for the 
length/verbosity. Also, as a general disclaimer, I realize that you are all 
experts; in many of my arguments, I occaisionally restate certain concepts that 
I know you are all intimately aware of to frame the argument. Corrections are, 
as always, welcome.

> 
> To put what Alex wrote in a somewhat different way, I'd say that the
> tension here is between explicit configuration (as one finds today in,
> e.g., the Maven world) and implicit configuration (IoC).

Just a small nit: IoC can also be explicit, its just that the explicitness is 
decoupled from the module, and controlled by another party, allowing for more 
flexibility in the assembled system.

> Both approaches
> are important.  The former is typical of standalone Java SE applications
> while the latter is typical of Java EE applications, though the two
> approaches are often intermixed.

I agree they are certainly intermixed elements of a system, but I’d also argue 
IoC is pervasive in SE applications as well (e.g. inclusion of 330 and 250 in 
SE are examples of a desire for SE usage). I can’t refute that it has greater 
usage in EE, since its part of the spec, and thus effectively every EE 
application. 

I think a better use case categorization of this problem is static linkage vs 
dynamic invocation. In static linkage an explicit symbol mapping resolved by 
the language itself is ideal as it avoids ambiguity, and by definition is 
static. On the other hand with dynamic invocation it’s common for the caller to 
utilize introspection and discovery as part of the natural flow of executing a 
dynamic call. Resolving ambiguity is not an issue in this case, since it is 
already handled by the caller as part of introspection.

> 
> What we have in the design today seems to support the explicit approach
> pretty well, but we're still trying to figure out how best to support the
> implicit approach.

(Thanks for trying to address this concern!)

> 
> If I understand correctly, your view of the present proposal is that:
> 
> (1) It induces too much boilerplate, requiring developers to write
> `exports dynamic P` for every single package `P` that's subject
> to reflection by a framework, and

That’s an accurate summary of this point. To the user they have already 
expressed their intention by including an annotation on their class, or 
expressing it in configuration. This requires that they restate that intention, 
in another area. It’s also a potential source of errors/confusion if a user 
unintentionally expresses an inconsistency. 
> 
> (2) It weakens encapsulation too much, by making the types in such a
> package available for reflection at run time by any module in the
> system.

Sorry for the confusion, what I was trying to say on this point was a bit 
different. What I was trying to say was:

(2') It weakens encapsulation by forcing the introduction of exports 
introducing potential conflicts that break applications. 

As an example, assume I have three modules with classloader-per-module 
isolation (A, B, and Victim)

- A exports foo, and has a non-exported package “bar"

- B exports bar

- Victim has a module-info with requires A; requires B

Now A decides to use IoC on some of its classes in bar, so it’s definition is 
changed to:
{ exports foo; exports dynamic bar; }

Since exports dynamic is internally a normal export at runtime, module 
resolution fails when loading Victim, because its now including a duplicate 
package, even though A had no intention of publishing its internal bar package 
for linkage.

Qualified exports could in theory address this problem, but they are 
problematic in a dynamic environment since the module is simply not in a 
position to know all of the various modules which would/could enhance it. 
Notably, a key aspect of IoC is that modules are decoupled. For example, a 
container might employ dynamic selection of multiple versions of the code 
performing the enhancement, and/or it might segment its implementation into 
multiple implementation modules. Additionally the code behind a module that 
uses qualified exports is less reusable, and has to be updated according to 
everything which may or may not reflectively access it. Ultimately you would 
end up needing to dynamically generate them as you mention later on, but there 
are still problems there as well (I’ll expend further below).

> 
> These observations lead to your suggestion to allow declarative module
> boundaries to be overridden by "trusted" framework code.  It's far from
> clear how to define such a facility in a way that would still allow us to
> achieve one of our primary goals, namely strong encapsulation, i.e., the
> ability of the author of a module to declare which types are accessible
> by other components, and which are not.

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Peter Levart]

Hi,

I have a question about the following ...

On 07/13/2016 11:47 PM, mark.reinh...@oracle.com wrote:

To point (2), if some packages in a user module need to be exported for
reflection at run time, and a container wishes to ensure that only select
"trusted" framework modules can access the types in those packages, then
that's already expressible today.  We can also ensure that the set of
packages exported by a module is the same whether it's used standalone
on Java SE versus inside a container, which as you observe elsewhere in
this thread [1] could be problematic.

Suppose, e.g., we have an application module that's written against JPA,
rather than any specific JPA implementation, and exports the package
containing its entity classes for reflection at run time:

 module com.foo.data {
 requires java.persistence;
 exports dynamic com.foo.data.model;
 }

When used standalone, outside of a container, this module will export the
package containing its entity classes for reflection at run time.  The
classes will be accessible to every other module, but from a security and
integrity standpoint we assume that whoever invokes the run-time system,
i.e., whoever provides the command-line arguments to the `java` launcher
or its equivalent, is trusted to ensure that no adversarial modules are
present.

When used inside a container, the container already has the power to
prevent an adversarial module from accessing the module's entity classes.
That's because we expect containers to load every application into a
unique layer [2], and a container can rewrite module descriptors when
configuring a layer.  This is nothing to be ashamed of -- we fully expect
it to become a common practice.

If the container is set up to provide, e.g., Hibernate to this particular
application, then it could narrow the accessibility of the entity classes
by rewriting the above module declaration to refine the `exports dynamic`
directive:

 module com.foo.data {
 requires java.persistence;
 exports dynamic com.foo.data.model
  to hibernate.core, hibernate.entitymanager;
 }

(This is one of the very few use cases for qualified dynamic exports.)

Whether standalone or in a container the same set of packages is exported
by the module; the only difference is that, inside the container, the
exports are qualified.


What additional metadata does container need to rewrite a module 
descriptor like in above example? Does it need the whole set of exports 
that replace existing dynamic exports? Or only the target modules that 
it "attaches" to all unqualified dynamic exports? If the later, then 
what does container do if one wants to rewrite only a selection of 
unqualified dynamic exports to target one set of modules (for example an 
IoC implementation) and  another (possibly overlapping) selection of 
exports to target some other set of modules (for example a JPA 
implementation)? Does it give up and "attaches" all targets to all 
unqualified dynamic exports ?


I think something is missing here. A kind of hook to identify or "tag" a 
set of unqualified dynamic exports so that it can be located by the 
container.


Regards, Peter

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[mark . reinhold]

Jason -- thanks for your feedback on this topic.

To put what Alex wrote in a somewhat different way, I'd say that the
tension here is between explicit configuration (as one finds today in,
e.g., the Maven world) and implicit configuration (IoC).  Both approaches
are important.  The former is typical of standalone Java SE applications
while the latter is typical of Java EE applications, though the two
approaches are often intermixed.

What we have in the design today seems to support the explicit approach
pretty well, but we're still trying to figure out how best to support the
implicit approach.

If I understand correctly, your view of the present proposal is that:

  (1) It induces too much boilerplate, requiring developers to write
  `exports dynamic P` for every single package `P` that's subject
  to reflection by a framework, and

  (2) It weakens encapsulation too much, by making the types in such a
  package available for reflection at run time by any module in the
  system.

These observations lead to your suggestion to allow declarative module
boundaries to be overridden by "trusted" framework code.  It's far from
clear how to define such a facility in a way that would still allow us to
achieve one of our primary goals, namely strong encapsulation, i.e., the
ability of the author of a module to declare which types are accessible
by other components, and which are not.  I'd therefore like to explain
and explore how the above issues can be addressed with the present
design.

  * * *

To point (1), we all know that the most common way for developers to
write Java code today is with a rich and powerful IDE.  These tools
already have plenty of built-in cleverness for generating POJO classes,
deriving precise `import` directives, and ameliorating other kinds of
boilerplate.  I don't think it would be at all a stretch for such tools
to generate precise `exports dynamic` directives on demand, based upon
the presence of IoC-style annotations, and maintain their consistency
over time.  Just as precise `import` directives in class and interface
declarations document dependences upon specific types, so precise
`exports dynamic` directives in a module declaration would document
the exposition of specific types for reflection at run time.

If we think it likely that some modules will need to export dozens or
hundreds of packages, leading to extremely long module declarations, then
one possible refinement would be to allow a wildcard: `exports dynamic *`
would export all of a module's packages for reflection at run time.  This
would likely be straightforward.

  * * *

To point (2), if some packages in a user module need to be exported for
reflection at run time, and a container wishes to ensure that only select
"trusted" framework modules can access the types in those packages, then
that's already expressible today.  We can also ensure that the set of
packages exported by a module is the same whether it's used standalone
on Java SE versus inside a container, which as you observe elsewhere in
this thread [1] could be problematic.

Suppose, e.g., we have an application module that's written against JPA,
rather than any specific JPA implementation, and exports the package
containing its entity classes for reflection at run time:

module com.foo.data {
requires java.persistence;
exports dynamic com.foo.data.model;
}

When used standalone, outside of a container, this module will export the
package containing its entity classes for reflection at run time.  The
classes will be accessible to every other module, but from a security and
integrity standpoint we assume that whoever invokes the run-time system,
i.e., whoever provides the command-line arguments to the `java` launcher
or its equivalent, is trusted to ensure that no adversarial modules are
present.

When used inside a container, the container already has the power to
prevent an adversarial module from accessing the module's entity classes.
That's because we expect containers to load every application into a
unique layer [2], and a container can rewrite module descriptors when
configuring a layer.  This is nothing to be ashamed of -- we fully expect
it to become a common practice.

If the container is set up to provide, e.g., Hibernate to this particular
application, then it could narrow the accessibility of the entity classes
by rewriting the above module declaration to refine the `exports dynamic`
directive:

module com.foo.data {
requires java.persistence;
exports dynamic com.foo.data.model
 to hibernate.core, hibernate.entitymanager;
}

(This is one of the very few use cases for qualified dynamic exports.)

Whether standalone or in a container the same set of packages is exported
by the module; the only difference is that, inside the container, the
exports are qualified.

  * * *

To 

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Alan Bateman]

On 12/07/2016 10:28, Andrew Dinn wrote:


:
I think I need to ask you to clarify this as it doesn't seem to
recognise the point I was making. Of course, that may well indicate that
I have failed to understand the precise behaviour of exports dynamic.

Let us assume Module M exports dynamic P, where P is a package which
contains a public class C_pub and a non-public class C_pri (let's say it
is package-protected).

My understanding is that this means that:

Java source files for classes not in module M which import these classes
or reference them by name will suffer a compile-time error.
Correct. Also an error (IllegalAccessError) at run-time if you create 
the bytecode by other means (or maybe dropping the public modifier and 
not re-compiling the compiler for example).




References to the corresponding instances of class Class for C_pub or
C_pri may nevertheless be obtained by methods of classes not in Module M
at runtime (e.g. using a classloader lookup by name).
Correct. This is visibility and is not changed (try Class.forName to 
load C_Pri with JDK 8 and you'll see the same thing).




References to members of these classes (methods or fields) may be
obtained by methods of classes not in Module M (e.g. by using the public
API of Class).
Correct. However if you attempt access (Method::invoke, Field::get, 
Constructor::newInstance) then it will fail with IllegalAccessException 
(exactly the same as JDK 8 and older).




Calls to setEnabled(true) on those members will succeed even when the
calling method is not in class M (assuming the call is not invalidated
by the usual (mon-module) security restrictions)
I assume you mean setAccessible(true), the sledge hammer that breaks the 
door down by suppressing Java Language access checks. In this case, 
package P is exported, and so setAccessible(true) will succeed.




The latter is so irrespective of whether we are referring to class C_pub
or C_pri and irrespective of whether the member of C_pub or C_pri in
question is public or non-public.
If a library or framework really needs to get at types or members that 
aren't accessible then this is what it will typically too. There are 
sometimes better solutions with Lookup objects but maybe the discussion 
will go there. For now I assume we need to establish the basics.


-Alan

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Andrew Dinn]

On 11/07/16 12:02, Alan Bateman wrote:
>> That option may well be the status quo as of JDK8 but with JDK9 it is
>> the status quo in a changed world. Firstly, this so-called status quo
>> significantly undermines the point and utility of Jigsaw since
>> effectively it negates it's presence at runtime for large parts of the
>> code base. So, you get compile-time checking but you don't get any
>> runtime enforcement.

> It means that public types in these packages are accessible to other
> components. Non-public types/members would not be accessible of course,
> at least not without suppressing access checks (= setAccessible when
> using core reflection).

I think I need to ask you to clarify this as it doesn't seem to
recognise the point I was making. Of course, that may well indicate that
I have failed to understand the precise behaviour of exports dynamic.

Let us assume Module M exports dynamic P, where P is a package which
contains a public class C_pub and a non-public class C_pri (let's say it
is package-protected).

My understanding is that this means that:

Java source files for classes not in module M which import these classes
or reference them by name will suffer a compile-time error.

References to the corresponding instances of class Class for C_pub or
C_pri may nevertheless be obtained by methods of classes not in Module M
at runtime (e.g. using a classloader lookup by name).

References to members of these classes (methods or fields) may be
obtained by methods of classes not in Module M (e.g. by using the public
API of Class).

Calls to setEnabled(true) on those members will succeed even when the
calling method is not in class M (assuming the call is not invalidated
by the usual (mon-module) security restrictions)

The latter is so irrespective of whether we are referring to class C_pub
or C_pri and irrespective of whether the member of C_pub or C_pri in
question is public or non-public.

I'd be grateful if you could confirm or contradict any of these points.

regards,


Andrew Dinn
---
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Alan Bateman]

On 11/07/2016 10:41, Andrew Dinn wrote:


:
I don't think there is any confusion here other that that you have
failed to note an important part of what is being asked for and, in
consequence, recognise why that request was made. Jason, Paul and I all
said we would like to see some sort of privileged version of what you
are proposing.

Alex has engaged with Jason on his comments, I'm sure Mark will comment too.



'exports dynamic' is not a privileged relaxation.

Correct and I don't think anyone has suggested otherwise.



That option may well be the status quo as of JDK8 but with JDK9 it is
the status quo in a changed world. Firstly, this so-called status quo
significantly undermines the point and utility of Jigsaw since
effectively it negates it's presence at runtime for large parts of the
code base. So, you get compile-time checking but you don't get any
runtime enforcement.
It means that public types in these packages are accessible to other 
components. Non-public types/members would not be accessible of course, 
at least not without suppressing access checks (= setAccessible when 
using core reflection).


-Alan

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Andrew Dinn]

On 09/07/16 22:22, Alan Bateman wrote:
> Hence the `exports dynamic` proposal. There is a lot of confusion in
> this thread and it might be useful if someone could try out a scenario
> with an injectable constructor or method on a type in an otherwise
> non-exported package. That might help get the discussion back on track
> and get on to discussions or proposals on usability (for example).

I don't think there is any confusion here other that that you have
failed to note an important part of what is being asked for and, in
consequence, recognise why that request was made. Jason, Paul and I all
said we would like to see some sort of privileged version of what you
are proposing. 'exports dynamic' is not a privileged relaxation. It is a
wholesale removal of Jigsaw's runtime control from whatever modules
containers might need to operate on (whether in the JDK runtime or in
application deployments).

That option may well be the status quo as of JDK8 but with JDK9 it is
the status quo in a changed world. Firstly, this so-called status quo
significantly undermines the point and utility of Jigsaw since
effectively it negates it's presence at runtime for large parts of the
code base. So, you get compile-time checking but you don't get any
runtime enforcement.

Secondly, in consequence, it constitutes a significant risk because it
makes it very easy for developers to believe they have secured private
information and functionality while requiring the middleware layer many
of them depend on to prejudice that security across the board. A
mechanism enabling restricted relaxation of the constraints on
reflective access would allow trusted tools and libraries to continue to
do what they have always been able to do while still providing the
secure encapsulation that is the main point of having Jigsaw.

You are right in suggesting that the ability to manage layers means that
middleware can actually configure more controlled access to non-public
reflection than is offered by using 'exports dynamic'. I am currently
looking into exactly that same option for my agent code and it may well
be adequate to achieving the desired 'controlled access'. However, like
Paul (to whom your comments above were posted) I'm also not convinced
that this is something that should be pushed onto developers of
middleware or agents.

Implementing module linling via the Layer API is definitely going to be
cumbersome. It is also going to be difficult to do efficiently in a
dynamic environment where deployments are commonly removed and
redeployed while the container continues running, potentially with
different requirements for linkage to container services. Redeployment
will not just require relinking the component immediately redeployed but
also dependent components which need stopping and restarting. If module
rewriting is going to be the only way to do this then it is going to
have to be extremely efficient. Implementing this once, efficiently in
the module system sounds to me like a much better approach.

regards,


Andrew Dinn
---
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Xavi Miró Guerrero]

Hi,

   in my opinion, the problem here is that there are two types of use 
cases that need a different "default" behavior with respect to exporting 
packages in runtime and if we decide to use the current behavior, the 
use cases that need to export everything by default in runtime can lead 
to cumbersome configuration, because the packages must be exported one 
by one. If we had a way to specify that we export in runtime all the 
packages (with "ALL" or "*"), then in one single line we would be able 
to express what we want to do. For instance:


..
exports dynamic ALL to java.ee;
exports dynamic ALL to org.springframework.core;
..

or if it's necessary:

..
exports dynamic ALL;
..

  Although my projects' use cases wouldn't need to export in runtime 
many packages, I can imagine an application with dependency injection, 
where the majority of classes have DI annotations (like "@Inject") that 
must be read in runtime by the DI framework. As the application is built 
by assembling a lot of classes annotated with this framework's 
annotation types, the packages needed to be exported could be a lot. 
Thus, exporting all of the packages in runtime can be cumbersome and 
having a one line export statement can be useful.


  The suggestion of Simon to reverse the runtime export behavior is 
interesting, but I'm not sure that this must be the default. I 
personally prefer that the packages are unexported (even in runtime) by 
default and specify what to export (and to whom, if possible, with the 
qualified version of it). For the use cases where all the packages must 
be exported in runtime, or when there are too many to be specified 
manually, a "ALL" or "*" wildcard would help.


  Best regards,

   Xavi

On 09.07.2016 23:07, Simon Nash wrote:

Paul Benedict wrote:

For those who are still supporters of preventing non-exported types from
being reflected, I think a compromise can still be found, but it's 
not in
this proposal. Here are two other alternatives I hope the EG will 
consider:


1) Introduce a new permission type to allow non-exported types to be
reflected. I don't find it acceptable the module gets to dictate what 
can't
be reflected. I believe this should be controlled and configured 
externally

-- certainly not the module itself.

2) Allow layers to control if non-exported types can be reflected. 
Perhaps
the JDK sets its own layers to "false", but Containers and what they 
deploy
can be separately configured, each. For example, maybe WebLogic won't 
allow
itself to have its non-exported types reflected, but if each EAR gets 
its

own layers, I could configure WebLogic to allow me to reflect everything
within the EAR.

PS: I don't see #1 and #2 to be mutually exclusive.

Cheers,
Paul

I would like to propose another possible compromise that might satisfy 
both

of the following requirements that have been expressed on this thread:

1) My module should completely encapsulate some internal classes and make
   them invisible to the outside world

2) My module or library needs reflective access to internal classes in
   other modules in order to function correctly

At present, a module can mark its packages as:
 exported at compile time and runtime
 exported only at runtime via reflection
 never exported (the default)

Instead, these options could change to:
 exported at compile time and runtime
 exported at runtime only via reflection (the default)
 never exported (hidden)

This would allow modules to mark certain packages as "hidden" if these
packages contain internal methods or fields that should be never be
accessible externally under any circumstances.  The default would be
to allow reflection by other modules or libraries that need such access.

 Simon

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Alan Bateman]

On 09/07/2016 12:22, Jochen Theodorou wrote:



ok, let us assume I want to write a library that does what reflection 
can do today using bytecode generation. And let us assume I write a 
library, that will use this reflective module to call from private API 
to private API. How do we configure the two modules and what, besides 
the bytecode generation and the api, does the reflective library have 
to do (layers for example)? Would that have the same power as core 
reflection as far as the module system is concerned?
Just so I understand. There is code in module m1 that uses this library 
to generate code to access something in module m2, is that right? Also 
when you say "private API" then I think you mean the API is in a package 
that is not exported by m2.


Assuming this is the scenario then I would expect the generated code to 
fail because the the API is not accessible outside of m2. That is, the 
code that the library generates (into m1?) will throw 
IllegalAccessError. This is no different of course from what you have 
with JDK 8 and older. Ignoring modules, if the secret API in m2 is a 
non-public class then it's inaccessible to code in m1.


Module layers might be too much to bring into the discussion at this 
point, esp. if m1 and m2 are in different layers with different class 
loaders and you have to understand the visibility before getting to 
accessibility.


As regards doing the equivalent with core reflection then it would be 
exactly the same. Code in m1 using core reflection in an attempt to get 
at the "private API" in m2 will fail with IllegalAccessException 
(setAccessible is also too much to being into the discussion at this point).



:

Oh... and let us assume there are two modules using this reflective 
module configured to be able to access their hidden API using the 
reflective library. Would this automatically allow the two libraries 
to call into the other hidden library?
If the so-called hidden API is public and m2 exports the package with 
that API to its friend m1 then it should work. I'm assuming here that 
the library generating the code is generating it into m1. If it's 
generating it into another module then it won't work of course, at least 
not unless m1 exports the package (maybe reflectively) to that module.


My guess is that your mail will have follow-on discussion so it might be 
better to start a new thread on jigsaw-devv so avoid it getting lost in 
the #ReflectiveAccessToNonExportedTypes discussion.


-Alan.

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Paul Benedict]

Alan, but that is extra configuration in the module, as I have pointed out.
The whole idea I have to opt someone into fuller reflection is taking
control too far. I'm also advocating this is not the responsibility of the
module to control reflection. What I have proposed using the Security
Manager allows people to restrict the runtime if they do so desire and
allow reflection to continue unhindered as-is otherwise. Trying out
"dynamic" doesn't address the design except confirming it works according
to your intent. I'm complaining (respectfully) about the intent and the
restriction as the default behavior.

On Jul 9, 2016 4:22 PM, "Alan Bateman"  wrote:

> On 09/07/2016 21:28, Paul Benedict wrote:
>
> The argument I'm making is not just someone really wants to do this, but
>> that many people won't settle without it. Reflection has always been the
>> tool to dynamically achieve what the Java language can't always express
>> statically. IoC is built on the notion that language boundaries can and
>> should be broken to achieve magic-like behavior like injecting. Look all
>> over the EE spec and see how injection doesn't care what visibility
>> modifier you use... private methods and private fields are just as readable
>> and writable like public counterparts. Nothing wrong here, nothing broken
>> either.
>>
>> Hence the `exports dynamic` proposal. There is a lot of confusion in this
> thread and it might be useful if someone could try out a scenario with an
> injectable constructor or method on a type in an otherwise non-exported
> package. That might help get the discussion back on track and get on to
> discussions or proposals on usability (for example).
>
> -Alan.
>

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Alan Bateman]

On 09/07/2016 21:28, Paul Benedict wrote:

The argument I'm making is not just someone really wants to do this, 
but that many people won't settle without it. Reflection has always 
been the tool to dynamically achieve what the Java language can't 
always express statically. IoC is built on the notion that language 
boundaries can and should be broken to achieve magic-like behavior 
like injecting. Look all over the EE spec and see how injection 
doesn't care what visibility modifier you use... private methods and 
private fields are just as readable and writable like public 
counterparts. Nothing wrong here, nothing broken either.


Hence the `exports dynamic` proposal. There is a lot of confusion in 
this thread and it might be useful if someone could try out a scenario 
with an injectable constructor or method on a type in an otherwise 
non-exported package. That might help get the discussion back on track 
and get on to discussions or proposals on usability (for example).


-Alan.

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Simon Nash]

Paul Benedict wrote:

For those who are still supporters of preventing non-exported types from
being reflected, I think a compromise can still be found, but it's not in
this proposal. Here are two other alternatives I hope the EG will consider:

1) Introduce a new permission type to allow non-exported types to be
reflected. I don't find it acceptable the module gets to dictate what can't
be reflected. I believe this should be controlled and configured externally
-- certainly not the module itself.

2) Allow layers to control if non-exported types can be reflected. Perhaps
the JDK sets its own layers to "false", but Containers and what they deploy
can be separately configured, each. For example, maybe WebLogic won't allow
itself to have its non-exported types reflected, but if each EAR gets its
own layers, I could configure WebLogic to allow me to reflect everything
within the EAR.

PS: I don't see #1 and #2 to be mutually exclusive.

Cheers,
Paul


I would like to propose another possible compromise that might satisfy both
of the following requirements that have been expressed on this thread:

1) My module should completely encapsulate some internal classes and make
   them invisible to the outside world

2) My module or library needs reflective access to internal classes in
   other modules in order to function correctly

At present, a module can mark its packages as:
 exported at compile time and runtime
 exported only at runtime via reflection
 never exported (the default)

Instead, these options could change to:
 exported at compile time and runtime
 exported at runtime only via reflection (the default)
 never exported (hidden)

This would allow modules to mark certain packages as "hidden" if these
packages contain internal methods or fields that should be never be
accessible externally under any circumstances.  The default would be
to allow reflection by other modules or libraries that need such access.

 Simon

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Paul Benedict]

The argument I'm making is not just someone really wants to do this, but
that many people won't settle without it. Reflection has always been the
tool to dynamically achieve what the Java language can't always express
statically. IoC is built on the notion that language boundaries can and
should be broken to achieve magic-like behavior like injecting. Look all
over the EE spec and see how injection doesn't care what visibility
modifier you use... private methods and private fields are just as readable
and writable like public counterparts. Nothing wrong here, nothing broken
either.

The whole notion that people would turn to Containers to jailbreak a
feature smells of a poor design. I'd say it's analogous to how many
developers abandoned EJB 2 for Spring until EJB 3 spec leads finally
acknowledged what developers really wanted. I think the reaction to this
restriction is, in my opinion, going to lead to similar behavior of a mass
hunt and development for a better solution.

I really don't want to turn to Containers to escape this restriction. I
prefer the JDK correctly solve it using the Security Manager to prevent an
incongruous solution with the past. To be congruent should be about how the
Security Manager can be finely turned to restrict reflection on
non-exported types. Oracle can use it to protect JDK internals (if Oracle
wants to), but the world can go on doing the reflection it depends on now
without more (module) configuration I prefer less configuration, and
more convention, unless I have a security reason not to.

And what about non-EE containers using injection? I suppose they will need
a custom framework to make everything reflectable -- maybe Spring will help
out here to universalize a solution? I don't know. But I do know I don't
like controlling how reflection acts through a module descriptor. That's
clearly a responsibility of the Security Manager and this kind of external
control should continue to be built upon.

On Jul 9, 2016 3:01 AM, "Alan Bateman"  wrote:

On 08/07/2016 22:42, Paul Benedict wrote:

:
>
>
> 2) Allow layers to control if non-exported types can be reflected. Perhaps
> the JDK sets its own layers to "false", but Containers and what they deploy
> can be separately configured, each.
>
>
> A container or anything else doing dynamic configuration today provides
the module finders and so gets an opportunity to re-write module
descriptors if it really wants to. So if someone really wants every module
in the configuration to export every package then it is possible. It's a
bit of effort of course but it seems better than introducting
inconsistencies into how core reflection does access checks vs. the VM and
method handles.

-Alan

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Simon Nash]

Alan Bateman wrote:

On 09/07/2016 09:57, Simon Nash wrote:



Is it really a good idea to encourage packages that currently use 
reflection
to access non-exported private fields (an official part of the Java 
API) to

change to using internal APIs?  This seems like a step backwards to me.
I didn't suggest that. I was just pointing out that some of the 3rd 
party serialization libraries that we've come across are based on Unsafe 
and ReflectionFactory and the latter is specifically on critical 
internal API for that reason.


-Alan


Yes, I understand.  My point was that these APIs might in some cases
become a substitute for reflective access to non-exported private fields
if the latter is prohibited,

 Simon

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Jochen Theodorou]

On 09.07.2016 09:50, Alan Bateman wrote:
[...]

As regards "enjoy special power" then the only module that is known to
the VM and module system is "java.base". The java.base module, as you
probably know, is the core of the system. The java.base modules has the
VM, java.lang.**, the implementation of core reflection, method handles,
and everything else that make up the core runtime and APIs. The other 70
or so standard and JDK-specific modules [1] are just modules, no
different to user modules that are deployed on the application module path.

-Alan

[1] http://openjdk.java.net/jeps/200


ok, let us assume I want to write a library that does what reflection 
can do today using bytecode generation. And let us assume I write a 
library, that will use this reflective module to call from private API 
to private API. How do we configure the two modules and what, besides 
the bytecode generation and the api, does the reflective library have to 
do (layers for example)? Would that have the same power as core 
reflection as far as the module system is concerned?


Since core reflection is in java.base, there is the assumption it enjoys 
special powers with regards to the module system. And that is especially 
because something like the layer system is still to unclear.


Oh... and let us assume there are two modules using this reflective 
module configured to be able to access their hidden API using the 
reflective library. Would this automatically allow the two libraries to 
call into the other hidden library?


bye Jochen

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Alan Bateman]

On 09/07/2016 09:57, Simon Nash wrote:



Is it really a good idea to encourage packages that currently use 
reflection
to access non-exported private fields (an official part of the Java 
API) to

change to using internal APIs?  This seems like a step backwards to me.
I didn't suggest that. I was just pointing out that some of the 3rd 
party serialization libraries that we've come across are based on Unsafe 
and ReflectionFactory and the latter is specifically on critical 
internal API for that reason.


-Alan

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Simon Nash]

Alan Bateman wrote:

On 09/07/2016 08:46, Simon Nash wrote:


I think this a very important point.  If someone wanted to reimplement
Java serialization (java.io.ObjectOutputStream, etc.) as an external 
library

(com.foo.ObjectOutputStream, etc.), the new restrictions on reflective
access in JDK 9 would prevent this.
If there are types in non-exported packages in the serial form then it 
could be an issue. Some serialization libraries are based on Unsafe and 
sun.reflect.ReflectionFactory, both "critical internal APIs" that 
continue to be available via the jdk.unsupported module. More on this in 
JEP 260 [1].


-Alan

[1] http://openjdk.java.net/jeps/260


Is it really a good idea to encourage packages that currently use reflection
to access non-exported private fields (an official part of the Java API) to
change to using internal APIs?  This seems like a step backwards to me.

 Simon

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Alan Bateman]

On 09/07/2016 08:46, Simon Nash wrote:


I think this a very important point.  If someone wanted to reimplement
Java serialization (java.io.ObjectOutputStream, etc.) as an external 
library

(com.foo.ObjectOutputStream, etc.), the new restrictions on reflective
access in JDK 9 would prevent this.
If there are types in non-exported packages in the serial form then it 
could be an issue. Some serialization libraries are based on Unsafe and 
sun.reflect.ReflectionFactory, both "critical internal APIs" that 
continue to be available via the jdk.unsupported module. More on this in 
JEP 260 [1].


-Alan

[1] http://openjdk.java.net/jeps/260

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Alan Bateman]

On 08/07/2016 22:42, Paul Benedict wrote:


:

2) Allow layers to control if non-exported types can be reflected. Perhaps
the JDK sets its own layers to "false", but Containers and what they deploy
can be separately configured, each.


A container or anything else doing dynamic configuration today provides 
the module finders and so gets an opportunity to re-write module 
descriptors if it really wants to. So if someone really wants every 
module in the configuration to export every package then it is possible. 
It's a bit of effort of course but it seems better than introducting 
inconsistencies into how core reflection does access checks vs. the VM 
and method handles.


-Alan

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Simon Nash]

Jason Greene wrote:


There is also a disparity here that the JDK itself doesn’t require you to 
export packages (e.g. I don’t need it for Java serialization). Now I realize 
that there is an effort underway to de-privilege modules, but I suspect that a 
portion of the JDK will continue to enjoy special power for precisely the same 
usability concerns that apply to frameworks / standards which extend the 
platform.


I think this a very important point.  If someone wanted to reimplement
Java serialization (java.io.ObjectOutputStream, etc.) as an external library
(com.foo.ObjectOutputStream, etc.), the new restrictions on reflective
access in JDK 9 would prevent this.

 Simon

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Gregg Wonderly]

The practical matter is simply that all of this name visibility business is 
just entirely the wrong thing to be managing “modules” around.  Instead, 
modules should be things that are more easily managed with tools that 
facilitate things that container systems will want to do.   Modularity as a 
design paradigm has nothing to do with namespace visibility.   Packages already 
provide name visibility and separation.  Keywords already provide restricted 
access unless you pry open the door with language features aimed specifically 
at solving the multiuse/reuse problem that public/internal/private restrictions 
create.  

For a long time I’ve set back and watched this project inching along with all 
kinds of people asking all the questions that I thought should be asked about 
why the EG was doing what they were doing.  I am really afraid i still don’t 
know why this hiding game is so interesting and valuable.  Is there really a 
giant problem that a majority of the Java community has now that this is going 
to just solve, and make developing faster and more readily managed?  
Versioning, and dynamic class replacement would be the kind of things that a 
module system could really provide to help software be more resilient to life 
cycle management over time.

We already have things like maven to help with dependency management.

We already have things like spring to help with dependency injection so that 
there are not hard bindings in code against specific versions.

We already have many different container systems that provide many different 
ways to manage software “pieces”.  The problem is that each one of them has a 
different paradigm and framework that one must program against.

Shouldn’t modularity really be about helping target more of those container 
systems implicitly instead of requiring explicit wrapper classes or other 
properties of the build system to make it possible to use the same module in 
multiple environments?

Why is this namespace hiding/exporting so important to Java developers?   Which 
Java developers need that level of isolation and why is it that everyone will 
have to deal with it, if those developers are not 90% of all developers 
benefiting?

Gregg Wonderly

> On Jul 8, 2016, at 4:42 PM, Paul Benedict  wrote:
> 
> For those who are still supporters of preventing non-exported types from
> being reflected, I think a compromise can still be found, but it's not in
> this proposal. Here are two other alternatives I hope the EG will consider:
> 
> 1) Introduce a new permission type to allow non-exported types to be
> reflected. I don't find it acceptable the module gets to dictate what can't
> be reflected. I believe this should be controlled and configured externally
> -- certainly not the module itself.
> 
> 2) Allow layers to control if non-exported types can be reflected. Perhaps
> the JDK sets its own layers to "false", but Containers and what they deploy
> can be separately configured, each. For example, maybe WebLogic won't allow
> itself to have its non-exported types reflected, but if each EAR gets its
> own layers, I could configure WebLogic to allow me to reflect everything
> within the EAR.
> 
> PS: I don't see #1 and #2 to be mutually exclusive.
> 
> Cheers,
> Paul
> 
> On Fri, Jul 8, 2016 at 4:16 PM, Jason Greene 
> wrote:
> 
>> 
>> On Jul 7, 2016, at 5:31 PM, Paul Benedict  wrote:
>> 
>> If this restriction stays (and I am really hoping it doesn't), my next
>> best hope is for Containers like WildFly, Tomcat, SpringBoot etc. to enable
>> me to do this. If the Layer has a hook into amending the Module Descriptor,
>> then I am hoping each Container will automatically set "dynamic" to each
>> non-exported package. I think this will be a highly requested and
>> sought-after feature.
>> 
>> 
>> That’s probably something we would do if this notion remains.
>> Unfortunately it means that you would then have different behavior for
>> standalone SE usage and container usage, making it harder to share code. It
>> also might introduce a conflict that wasn’t there before (e.g. split
>> package)
>> 
>> --
>> Jason T. Greene
>> WildFly Lead / JBoss EAP Platform Architect
>> JBoss, a division of Red Hat
>> 
>> 

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Paul Benedict]

For those who are still supporters of preventing non-exported types from
being reflected, I think a compromise can still be found, but it's not in
this proposal. Here are two other alternatives I hope the EG will consider:

1) Introduce a new permission type to allow non-exported types to be
reflected. I don't find it acceptable the module gets to dictate what can't
be reflected. I believe this should be controlled and configured externally
-- certainly not the module itself.

2) Allow layers to control if non-exported types can be reflected. Perhaps
the JDK sets its own layers to "false", but Containers and what they deploy
can be separately configured, each. For example, maybe WebLogic won't allow
itself to have its non-exported types reflected, but if each EAR gets its
own layers, I could configure WebLogic to allow me to reflect everything
within the EAR.

PS: I don't see #1 and #2 to be mutually exclusive.

Cheers,
Paul

On Fri, Jul 8, 2016 at 4:16 PM, Jason Greene 
wrote:

>
> On Jul 7, 2016, at 5:31 PM, Paul Benedict  wrote:
>
> If this restriction stays (and I am really hoping it doesn't), my next
> best hope is for Containers like WildFly, Tomcat, SpringBoot etc. to enable
> me to do this. If the Layer has a hook into amending the Module Descriptor,
> then I am hoping each Container will automatically set "dynamic" to each
> non-exported package. I think this will be a highly requested and
> sought-after feature.
>
>
> That’s probably something we would do if this notion remains.
> Unfortunately it means that you would then have different behavior for
> standalone SE usage and container usage, making it harder to share code. It
> also might introduce a conflict that wasn’t there before (e.g. split
> package)
>
> --
> Jason T. Greene
> WildFly Lead / JBoss EAP Platform Architect
> JBoss, a division of Red Hat
>
>

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Jason Greene]

> On Jul 7, 2016, at 5:31 PM, Paul Benedict  wrote:
> 
> If this restriction stays (and I am really hoping it doesn't), my next best 
> hope is for Containers like WildFly, Tomcat, SpringBoot etc. to enable me to 
> do this. If the Layer has a hook into amending the Module Descriptor, then I 
> am hoping each Container will automatically set "dynamic" to each 
> non-exported package. I think this will be a highly requested and 
> sought-after feature.

That’s probably something we would do if this notion remains. Unfortunately it 
means that you would then have different behavior for standalone SE usage and 
container usage, making it harder to share code. It also might introduce a 
conflict that wasn’t there before (e.g. split package)

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Jason Greene]

> On Jul 7, 2016, at 5:10 PM, Alex Buckley  wrote:
> 
> Hi Jason,

Hi Alex, 

Thank you for the thoughtful reply, I have included some debate below.

> 
> On 7/7/2016 1:17 PM, Jason Greene wrote:
>> I wanted to second Paul’s comments on jams-spec-comments[1], but with
>> some additional thoughts.
>> 
>> The proposal takes a step in the right direction by allowing a
>> runtime path to bypass access control. However, the fundamental issue
>> at play is that class visibility is being used as an access control
>> mechanism, and these concerns are really orthogonal notions. One of
>> Java’s most powerful historic capabilities is that it is fully
>> introspective and dynamic, and this has empowered a large ecosystem
>> of frameworks and platforms that have extended the language in novel
>> ways. All of which ultimately lead to it being a dominant server side
>> development platform.
>> 
>> A major commonality in modern programming models is the notion of
>> inversion of control. For those unaware, with IOC, a fundamental
>> notion is that the user defines code which is solely focused on a
>> particular concern, and it is completely unaware of other parts of
>> the system which later enhance that code. JSR 250 (part of SE),
>> demonstrates an example, the user simply needs to add a few
>> annotations, and their classes are dynamically augmented at runtime
>> with new behavior.  JSR 330 (also part of SE) is another, a container
>> of some sort is responsible for constructing classes that could be
>> anywhere in any module and injecting those instances based on some
>> set of rules into other classes’ private fields. This sort of thing
>> is very pervasive (most specs that make up Java EE, JPA, Spring,
>> JAXB, CXF, custom serialization frameworks, mock frameworks, etc).
>> Even the JDK itself needs this ability.
> 
> I call this the "School of Abstraction" form of modularity, which is about 
> hiding _values_, versus the "School of Encapsulation" form of modularity, 
> which is about hiding _names_.

I don’t view these as distinct concepts, but rather see them as interrelated 
and complimentary. Encapsulation is critical to inversion of control, because 
to achieve the decoupling that is required, you need well defined contracts 
throughout the system. Additionally, the decoupling and flexibility introduced 
naturally leads to the need to isolate the implementation and internal 
dependencies of what defines a module. So you certainly need name based 
filtering as well, the key difference is that you delegate the arrangement of 
components and modules, as well as specific behavior patterns to other parties 
that cooperatively define the end system. 

> 
> The Java language and VM, with their name-based rules for accessibility, have 
> long been in the Encapsulation school. IoC technologies, with their 
> abstraction over class instances, are in the Abstraction school. They had to 
> look outside the Java language and VM, to Core Reflection 
> (java.lang.reflect), to circumvent name-based accessibility (setAccessible).
> 
> In SE 9, the Java language and VM gain stronger name-based accessibility 
> (unexported package ==> inaccessible public types) without a corresponding 
> uplift in Core Reflection to let IoC technologies circumvent it. Thus, 
> tension between the two schools.

I argue that dynamic programming (reflection + setAccessible) was just as much 
a part of the language as static linking was, and it is one of the most 
powerful aspects of the language. Strict type safety and linkage has lead to 
less errors, and dynamic behavior has made it easy to build flexible systems 
and exceed the limitations of the language itself. You really have the best of 
both worlds in Java, and that is one of the main reasons I attribute to its 
success. 

I think it’s also important to note that dynamic systems which utilize 
inversion of control don’t suffer from the primary negatives of breaking 
encapsulation, since they aren’t statically defined against an implementation, 
but are rather generated/derived from it (e.g. don’t suffer from unexpected 
changes). There is of course always the associated security concerns with such 
power, but they aren’t unsolvable and most certainly worth the benefits (e.g. 
the whole ecosystem around Java that we have today).


> 
>> This brings us to the problem with the proposal. The expectation
>> AFAICT appears to be that the user defining the enhanced code knows
>> the identity of the module enhancing it (e.g. exports dynamic
>> com.foo.app.model to jpa). The module is simply not in a position to
>> know that just “jpa” requires access. There might be more than one
>> version of jpa which needs to be selected dynamically, or jpa might
>> be broken into multiple modules itself. It’s effectively bleeding
>> container/framework implementation details into the user defined
>> code.
> 
> 'exports dynamic' does not have to be qualified with a 'to' clause.

Right 

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Alan Bateman]

On 08/07/2016 13:46, Andrew Dinn wrote:


:
Jason addressed some of this in his post -- the nub (but definitely not
the total) of his points is in this paragraph

"This brings us to the problem with the proposal. The expectation AFAICT
appears to be that the user defining the enhanced code knows the
identity of the module enhancing it (e.g. exports dynamic
com.foo.app.model to jpa). The module is simply not in a position to
know that just “jpa” requires access. There might be more than one
version of jpa which needs to be selected dynamically, or jpa might be
broken into multiple modules itself. It’s effectively bleeding
container/framework implementation details into the user defined code."
I think the focus on qualified exports has been a bit of a distraction 
in this thread. It might have been clearer if the thread has started out 
without mentioning that possibility.



:

The major problem is the scale and difficulty of the legacy problem that
you are suggesting such a 'simple' fix for. There are already many, many
deployments which are not built the way you suggest they ought to be
rebuilt. It is very easy to say 'well sort them out' but that fails to
recognise the amount of work involved.

I don't recall anyone suggesting a simple fix. However, the so-called 
"legacy problem" is a great topic and something that you might want to 
spin off to its own thread. I'm not ignoring the rest of the mail, just 
noting that a lot of appears to be reliable configuration.


-Alan.

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Andrew Dinn]

On 08/07/16 10:16, Alan Bateman wrote:
>
> On 08/07/2016 09:39, Sander Mak wrote:
>> That is, define your module-info's, possibly containing dynamic
>> exports if your framework of choice does its magic through reflection.
>>
> Right and there is nothing in this proposal that is specific to core
> reflection either. The framework that I am using might be spinning
> bytecode or maybe it switches to using method handles when there is a
> waning gibbous moon - that's implementation detail that the user of the
> framework should be oblivious too.

Jason addressed some of this in his post -- the nub (but definitely not
the total) of his points is in this paragraph

"This brings us to the problem with the proposal. The expectation AFAICT
appears to be that the user defining the enhanced code knows the
identity of the module enhancing it (e.g. exports dynamic
com.foo.app.model to jpa). The module is simply not in a position to
know that just “jpa” requires access. There might be more than one
version of jpa which needs to be selected dynamically, or jpa might be
broken into multiple modules itself. It’s effectively bleeding
container/framework implementation details into the user defined code."

There are more complexities at stake here than you seem to recognise.
Jason goes on to point some of them out but I'll draw out one specific
aspect of it.

The major problem is the scale and difficulty of the legacy problem that
you are suggesting such a 'simple' fix for. There are already many, many
deployments which are not built the way you suggest they ought to be
rebuilt. It is very easy to say 'well sort them out' but that fails to
recognise the amount of work involved.

It is not a question of each app developer taking one or two jars and
rebuilding them with a module-info package. First of all there is the
complexity on the app server itself. An app server itself is built from
many components and employs many libraries a lot of them obtained from
3rd parties. In many cases those components allow for many alternative
combinations. The same issue then arises for the deployed applications.
They also frequently need to use many components in many different
combinations. That means it will require a lot of work to identify the
relevant dependencies and configure the necessary linkage without
exposing the wrong components to the wrong clients. But that's not the
worst of it.

The reason JBoss's app server developed its own module system many years
ago was precisely in order to manage the complexity of the combinatorics
involved in establishing this linkage. It's major purpose was to ensure
that most of the necessary linkage was established by the app server at
runtime using whatever services were found to have been configured. On
the deployment side Java EE annotations and legacy deployment
descriptors largely establish what linkage is required both from
application deployments into EE services and from EE services back into
application code (i.e. where IoC is required). The vast majority of this
linkage is established correctly and safely, with constrained access, by
the app server. Users know little or nothing of the details of how it is
established beyond having to employ those standard annotations or legacy
descriptors.

What you are asking for is for users to identify and reconstruct all
this information in advance of deployment in all possible combinations
that might occur at runtime. In many cases they won't even know how to
do this because the whole point of an app server is to reduce the amount
of work they need to do to link in services to simply adding annotations
or dropping an xml file into a deployment. That simplification is there
to allow EE developer to concentrate on business logic.

What you are asking fundamentally undermines one of the major benefits
of using EE servers. You have argued that users ought to take
responsibility for understanding the way their applications are linked
and be grateful that Jigsaw will now verify their choices. That's
completely contradicted by the historical success of app servers which
have done the precise opposite, i.e. taken this responsibility away from
programmers and managed the task on their behalf. Now a module system
which allows that to be done cleanly without exposing every capability
to all and sundry is a great idea and Jigsaw could be that. But your
assumption that end users will want and benefit from taking control of
this task is completely out of touch.

It is very telling that that, unlike Jigsaw, JBoss's module
implementation adopted a flexible model for linkage (export and import)
that prioritised establishment of dynamic linkage. Most importantly this
linkage can be configured external to the deployed artifacts and,
therefore, can be reconfigured without having to rebuild chains of
dependent artifacts. When there are so many links and so many of the
linked items are developed by different projects this is a critical
feature. This is one point where 

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Gunnar Morling]

Hi,

Putting my user's hat on, I'm interested in a solution which a) fulfils my
desire to hide types internal to my implementation as much as possible
(that's why I'm using a module system after all) and b) still is easy to
use.

Looking at the proposal, I feel adding *un-qualified* dynamic exports goes
against a) - every other module can access my internal types reflectively.
b) is only partly addressed, too - I need to list all my internal packages
(there may be many of them).

Adding *qualified* dynamic exports addresses a) (only selected modules may
access my internal types), but makes things worse for b) as I now need to
list all my internal packages *and* the modules meant to access them.
Tooling for generating the exports may help, but I feel that's dealing with
symptoms rather than addressing the cause (1).

Making internal types available to reflection by default as suggested
before in this thread does address a) not at all, but b) very much.

Personally, I find none of these solutions very compelling.

A solution that'd address a) and b) would be a way for allowing me as a
user to whitelist "trust-worthy modules" in one single spot.

I.e. a global descriptor of sorts where I can say "export foo.internal.* to
hibernate". I know Hibernate needs to use reflection, I'm ok with it doing
this, but I want to express that fact only once in a single place, and
using something like wildcards, for the sake of b). As the export is
qualified, and if needed also could be given on a more fine-grained level,
a) is addressed, too.

Such descriptor should live on the application (not module) level for the
sake of b) but also in order to allow for alignments with the specific
runtime environment. E.g. a third party module might have been compiled
with EclipseLink, while in my application I might want to use it with
Hibernate.

In container environments (Java EE), the container may create that
descriptor to address all the requirements of all the modules a specific
deployment is using to further simplify the user's life.

I think such approach would help with exposing internal types as less as
possible while still keeping things manageable for the user.

--Gunnar

(1) Btw. qualified exports with several targets (export foo to a, b) may be
tough to deal with when it comes to different tools involved. E.g. two
annotation processors, each aware of one such target requiring the export.
If that could be given in two distinct export statements, that'd help as
each tool than simply can add its entry rather than potentially updating an
existing one.



2016-07-08 11:16 GMT+02:00 Alan Bateman :

>
>
> On 08/07/2016 09:39, Sander Mak wrote:
>
>> That is, define your module-info's, possibly containing dynamic exports
>> if your framework of choice does its magic through reflection.
>>
>> Right and there is nothing in this proposal that is specific to core
> reflection either. The framework that I am using might be spinning bytecode
> or maybe it switches to using method handles when there is a waning gibbous
> moon - that's implementation detail that the user of the framework should
> be oblivious too.
>
> -Alan
>

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Alan Bateman]

On 08/07/2016 09:39, Sander Mak wrote:

That is, define your module-info's, possibly containing dynamic exports if your 
framework of choice does its magic through reflection.

Right and there is nothing in this proposal that is specific to core 
reflection either. The framework that I am using might be spinning 
bytecode or maybe it switches to using method handles when there is a 
waning gibbous moon - that's implementation detail that the user of the 
framework should be oblivious too.


-Alan

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Alan Bateman]

On 07/07/2016 23:31, Paul Benedict wrote:


:

If this restriction stays (and I am really hoping it doesn't), my next best
hope is for Containers like WildFly, Tomcat, SpringBoot etc. to enable me
to do this. If the Layer has a hook into amending the Module Descriptor,
then I am hoping each Container will automatically set "dynamic" to each
non-exported package. I think this will be a highly requested and
sought-after feature.

I'm curious why you think you need to do this. If I'm developing a 
library as a module then I'll export the API packages. If I'm using DI, 
or JPA, or making use of other frameworks, and where I'm putting 
annotations or have configuration that includes types in non-API 
packages, then it does mean I need to think about. I would at least 
expect the framework documentation, tutorials, examples, etc. to at 
least show me what I need to do when I'm using these frameworks in a 
module. Better still would be tools, maybe Maven plugins, to catch the 
cases like @Inject on a constructor/method in a non-exported package or 
an entity class in XML configuration where that class is in a 
non-exported package. Tooling that catches the opposite (needless 
exports) would be useful too. If there is cooperation from the 
frameworks, and if useful tooling emerges, then I don't think the 
proposal on the table is too bad.


-Alan

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Sander Mak]

> On 08 Jul 2016, at 10:04, Andrew Dinn  wrote:
> 
> Applying your
> logic to the status quo ante and deriving JDK9 will break the most
> significant body of code that is built on Java in a way that, as Jason
> argues, is /in practice/ impossible to undo. 

How so? The classpath continues to work on JDK9, and reflection works the same 
as before in the unnamed module (modulo some JDK internals that have been 
hidden, but that's not what we were talking about). If you then wish to move to 
modules, it only seems fair that you at least put some thought into what is 
encapsulated and what not. That is, define your module-info's, possibly 
containing dynamic exports if your framework of choice does its magic through 
reflection.

Of course, some developers may have been blissfully unaware that 
Hibernate/Spring/etc is using reflection on their code. I believe it's a good 
thing this dependency becomes explicit. 


Sander

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Andrew Dinn]

On 07/07/16 23:31, Paul Benedict wrote:
> It should be pointed out that the only reason IoC containers can succeed
> with setAccessible() is because developers commonly run without the
> Security Manager enabled. People who use IoC want to this circumvention on
> purpose. It's not an oversight -- it's intended. As far as I am concerned,
> if you're a module running inside of my application, I have every right to
> reflect into every you. That's my right, of course, unless I have
> explicitly turned on the Security Manager. So if I want the magic, I can
> have it. If I want to forbid it, I can but I do not buy into the
> argument the Module System is doing me a favor by preventing me (de facto)
> from reflecting into the non-exported types.

That's not quite correct. Another way IoC containers can succeed with
setAccessible() is for their developers to run with the
Security Manager very carefully and specifically enabled to allow their
own privileged code to use setAccessible(). When my agent runs inside
EAP or Wildfly it has to establish a security policy that allows it to
use reflection because that option is not granted to it by default.

Luckily, the use of a security policy for this purpose is something that
can be configured external to the code i.e. without having to rebuild,
redistribute and redeploy the applications, the container and the
enormous array of 3rd party components it depends on.

Well, no, I take that last point back. Of course, it happened by design
rather than by luck -- because it was driven by pragmatic, practical
concerns rather than a model of how applications ought to be written.

> If this restriction stays (and I am really hoping it doesn't), my next best
> hope is for Containers like WildFly, Tomcat, SpringBoot etc. to enable me
> to do this. If the Layer has a hook into amending the Module Descriptor,
> then I am hoping each Container will automatically set "dynamic" to each
> non-exported package. I think this will be a highly requested and
> sought-after feature.

It will also be a rather dangerous feature to enable. A more restricted
model for managing access would be far preferable.

regards,


Andrew Dinn
---
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Andrew Dinn]

Hi Alex,

Your rationale is very logical. The flaw I see in your argument is not
that it makes invalid inferences, rather that it is based on wrong
premises. Indeed, that flaw is visible in the result. Applying your
logic to the status quo ante and deriving JDK9 will break the most
significant body of code that is built on Java in a way that, as Jason
argues, is /in practice/ impossible to undo. That's not just EE servers
themselves that go down the pan. In consequence of breaking them you
break by the majority of code that is commercially and socially important.

In other words Jason's criticism is not an academic point about how
applications might use the capabilities offered by a JDK like JDK8 vs a
JDK like the JDK9 you are proposing. It's a practical point about how we
keep the ship afloat. His suggestion that some level of privileged
access be supported to allow a controlled continuation of the status quo
is a purely pragmatic suggestion to keep Java working and relevant.

n.b. the key word above is controlled. The current proposal to remedy
this issue at the very least undermines many of the benefits Jigsaw
clearly offers. What is worse is that it will confound the expectations
of users who rely on Jigsaw to manage access only to find that the
promise it offers is not in effect at runtime. I am very concerned that
this will lead to many serious security issues.

In sum, I believe your rejection of Jason's approach on the grounds of
logical coherence risks sinking the ship. At some point logic has to be
qualified and recast in the light of experience. What Jason is proposing
is neither illogical nor incoherent. It does, however, appear to me to
be absolutely necessary given the status quo.

regards,


Andrew Dinn
---
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander

On 07/07/16 23:10, Alex Buckley wrote:
> Hi Jason,
> 
> On 7/7/2016 1:17 PM, Jason Greene wrote:
>> I wanted to second Paul’s comments on jams-spec-comments[1], but with
>> some additional thoughts.
>>
>> The proposal takes a step in the right direction by allowing a
>> runtime path to bypass access control. However, the fundamental issue
>> at play is that class visibility is being used as an access control
>> mechanism, and these concerns are really orthogonal notions. One of
>> Java’s most powerful historic capabilities is that it is fully
>> introspective and dynamic, and this has empowered a large ecosystem
>> of frameworks and platforms that have extended the language in novel
>> ways. All of which ultimately lead to it being a dominant server side
>> development platform.
>>
>> A major commonality in modern programming models is the notion of
>> inversion of control. For those unaware, with IOC, a fundamental
>> notion is that the user defines code which is solely focused on a
>> particular concern, and it is completely unaware of other parts of
>> the system which later enhance that code. JSR 250 (part of SE),
>> demonstrates an example, the user simply needs to add a few
>> annotations, and their classes are dynamically augmented at runtime
>> with new behavior.  JSR 330 (also part of SE) is another, a container
>> of some sort is responsible for constructing classes that could be
>> anywhere in any module and injecting those instances based on some
>> set of rules into other classes’ private fields. This sort of thing
>> is very pervasive (most specs that make up Java EE, JPA, Spring,
>> JAXB, CXF, custom serialization frameworks, mock frameworks, etc).
>> Even the JDK itself needs this ability.
> 
> I call this the "School of Abstraction" form of modularity, which is
> about hiding _values_, versus the "School of Encapsulation" form of
> modularity, which is about hiding _names_.
> 
> The Java language and VM, with their name-based rules for accessibility,
> have long been in the Encapsulation school. IoC technologies, with their
> abstraction over class instances, are in the Abstraction school. They
> had to look outside the Java language and VM, to Core Reflection
> (java.lang.reflect), to circumvent name-based accessibility
> (setAccessible).
> 
> In SE 9, the Java language and VM gain stronger name-based accessibility
> (unexported package ==> inaccessible public types) without a
> corresponding uplift in Core Reflection to let IoC technologies
> circumvent it. Thus, tension between the two schools.
> 
>> This brings us to the problem with the proposal. The expectation
>> AFAICT appears to be that the user defining the enhanced code knows
>> the identity of the module enhancing it (e.g. exports dynamic
>> com.foo.app.model to jpa). The module is simply not in a position to
>> know that just “jpa” requires access. There might be more than one
>> version of jpa which needs to be selected dynamically, or jpa might
>> be broken into multiple modules itself. It’s effectively 

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Paul Benedict]

It should be pointed out that the only reason IoC containers can succeed
with setAccessible() is because developers commonly run without the
Security Manager enabled. People who use IoC want to this circumvention on
purpose. It's not an oversight -- it's intended. As far as I am concerned,
if you're a module running inside of my application, I have every right to
reflect into every you. That's my right, of course, unless I have
explicitly turned on the Security Manager. So if I want the magic, I can
have it. If I want to forbid it, I can but I do not buy into the
argument the Module System is doing me a favor by preventing me (de facto)
from reflecting into the non-exported types.

If this restriction stays (and I am really hoping it doesn't), my next best
hope is for Containers like WildFly, Tomcat, SpringBoot etc. to enable me
to do this. If the Layer has a hook into amending the Module Descriptor,
then I am hoping each Container will automatically set "dynamic" to each
non-exported package. I think this will be a highly requested and
sought-after feature.

Cheers,
Paul

On Thu, Jul 7, 2016 at 5:10 PM, Alex Buckley 
wrote:

> Hi Jason,
>
>
> On 7/7/2016 1:17 PM, Jason Greene wrote:
>
>> I wanted to second Paul’s comments on jams-spec-comments[1], but with
>> some additional thoughts.
>>
>> The proposal takes a step in the right direction by allowing a
>> runtime path to bypass access control. However, the fundamental issue
>> at play is that class visibility is being used as an access control
>> mechanism, and these concerns are really orthogonal notions. One of
>> Java’s most powerful historic capabilities is that it is fully
>> introspective and dynamic, and this has empowered a large ecosystem
>> of frameworks and platforms that have extended the language in novel
>> ways. All of which ultimately lead to it being a dominant server side
>> development platform.
>>
>> A major commonality in modern programming models is the notion of
>> inversion of control. For those unaware, with IOC, a fundamental
>> notion is that the user defines code which is solely focused on a
>> particular concern, and it is completely unaware of other parts of
>> the system which later enhance that code. JSR 250 (part of SE),
>> demonstrates an example, the user simply needs to add a few
>> annotations, and their classes are dynamically augmented at runtime
>> with new behavior.  JSR 330 (also part of SE) is another, a container
>> of some sort is responsible for constructing classes that could be
>> anywhere in any module and injecting those instances based on some
>> set of rules into other classes’ private fields. This sort of thing
>> is very pervasive (most specs that make up Java EE, JPA, Spring,
>> JAXB, CXF, custom serialization frameworks, mock frameworks, etc).
>> Even the JDK itself needs this ability.
>>
>
> I call this the "School of Abstraction" form of modularity, which is about
> hiding _values_, versus the "School of Encapsulation" form of modularity,
> which is about hiding _names_.
>
> The Java language and VM, with their name-based rules for accessibility,
> have long been in the Encapsulation school. IoC technologies, with their
> abstraction over class instances, are in the Abstraction school. They had
> to look outside the Java language and VM, to Core Reflection
> (java.lang.reflect), to circumvent name-based accessibility (setAccessible).
>
> In SE 9, the Java language and VM gain stronger name-based accessibility
> (unexported package ==> inaccessible public types) without a corresponding
> uplift in Core Reflection to let IoC technologies circumvent it. Thus,
> tension between the two schools.
>
> This brings us to the problem with the proposal. The expectation
>> AFAICT appears to be that the user defining the enhanced code knows
>> the identity of the module enhancing it (e.g. exports dynamic
>> com.foo.app.model to jpa). The module is simply not in a position to
>> know that just “jpa” requires access. There might be more than one
>> version of jpa which needs to be selected dynamically, or jpa might
>> be broken into multiple modules itself. It’s effectively bleeding
>> container/framework implementation details into the user defined
>> code.
>>
>
> 'exports dynamic' does not have to be qualified with a 'to' clause.
>
> To run reliably in a Java EE container, the user would have to not
>> qualify dynamic export, and also define an export on everything to
>> every module always. But then we run into a lot of boiler plate that
>> has to always be added by the user and is easily forgotten and/or
>> misconfigured for a very common use case. This runs counter to the
>> goals of many modern programming models, which seek to eliminate
>> boilerplate.
>>
>
> A long 'exports dynamic' list is precisely the tension between the two
> schools. Even if an IDE generates and maintains it, its presence is a
> reminder to the developer that his module's _internal packages_ are
> 

Re: Feedback on proposal for #ReflectiveAccessToNonExportedTypes

[Alex Buckley]

Hi Jason,

On 7/7/2016 1:17 PM, Jason Greene wrote:

I wanted to second Paul’s comments on jams-spec-comments[1], but with
some additional thoughts.

The proposal takes a step in the right direction by allowing a
runtime path to bypass access control. However, the fundamental issue
at play is that class visibility is being used as an access control
mechanism, and these concerns are really orthogonal notions. One of
Java’s most powerful historic capabilities is that it is fully
introspective and dynamic, and this has empowered a large ecosystem
of frameworks and platforms that have extended the language in novel
ways. All of which ultimately lead to it being a dominant server side
development platform.

A major commonality in modern programming models is the notion of
inversion of control. For those unaware, with IOC, a fundamental
notion is that the user defines code which is solely focused on a
particular concern, and it is completely unaware of other parts of
the system which later enhance that code. JSR 250 (part of SE),
demonstrates an example, the user simply needs to add a few
annotations, and their classes are dynamically augmented at runtime
with new behavior.  JSR 330 (also part of SE) is another, a container
of some sort is responsible for constructing classes that could be
anywhere in any module and injecting those instances based on some
set of rules into other classes’ private fields. This sort of thing
is very pervasive (most specs that make up Java EE, JPA, Spring,
JAXB, CXF, custom serialization frameworks, mock frameworks, etc).
Even the JDK itself needs this ability.


I call this the "School of Abstraction" form of modularity, which is 
about hiding _values_, versus the "School of Encapsulation" form of 
modularity, which is about hiding _names_.


The Java language and VM, with their name-based rules for accessibility, 
have long been in the Encapsulation school. IoC technologies, with their 
abstraction over class instances, are in the Abstraction school. They 
had to look outside the Java language and VM, to Core Reflection 
(java.lang.reflect), to circumvent name-based accessibility (setAccessible).


In SE 9, the Java language and VM gain stronger name-based accessibility 
(unexported package ==> inaccessible public types) without a 
corresponding uplift in Core Reflection to let IoC technologies 
circumvent it. Thus, tension between the two schools.



This brings us to the problem with the proposal. The expectation
AFAICT appears to be that the user defining the enhanced code knows
the identity of the module enhancing it (e.g. exports dynamic
com.foo.app.model to jpa). The module is simply not in a position to
know that just “jpa” requires access. There might be more than one
version of jpa which needs to be selected dynamically, or jpa might
be broken into multiple modules itself. It’s effectively bleeding
container/framework implementation details into the user defined
code.


'exports dynamic' does not have to be qualified with a 'to' clause.


To run reliably in a Java EE container, the user would have to not
qualify dynamic export, and also define an export on everything to
every module always. But then we run into a lot of boiler plate that
has to always be added by the user and is easily forgotten and/or
misconfigured for a very common use case. This runs counter to the
goals of many modern programming models, which seek to eliminate
boilerplate.


A long 'exports dynamic' list is precisely the tension between the two 
schools. Even if an IDE generates and maintains it, its presence is a 
reminder to the developer that his module's _internal packages_ are 
inspected at run time by _external forces_. I understand the viewpoint 
that says "So what? It's the Java way for frameworks to peek into user 
code!", and at the same time I appreciate the viewpoint that says 
"Explicit definitions of a module's content and surface are a good thing 
for long-term maintenance".



Additionally now the user is forced to define their visibility wider
than necessary, causing potential conflicts that would have otherwise
been avoided. Also any security benefit the access control check has
seems easily defeated at this point, since IIUC anyone can just
compile against a different definition removing the dynamic
modifier.


Point of order: you mention visibility but visibility is about class 
loading, and Jigsaw does not change that. We're talking about 
accessibility -- JLS 6.6 and JVMS 5.4.4.


Accessibility at compile time can't be easily defeated since changing 
someone else's module declaration to remove 'dynamic' is akin to 
changing someone else's type declaration to turn package->public -- 
simply not done.



One could address the boilerplate/usabilty issue by inverting this
mechanism from opt-in to opt-out (all packages are dynamic export
unless a restriction is specified). However the other visibility
issues aren’t really addressed.

I think the only way to truly solve this problem