> On Jul 13, 2016, at 4:47 PM, mark.reinh...@oracle.com wrote:
>
> Jason -- thanks for your feedback on this topic.
Hi Mark,
Thanks for you reply! My thoughts are inline. I apologize in advance for the
length/verbosity. Also, as a general disclaimer, I realize that you are all
experts; in many of my arguments, I occaisionally restate certain concepts that
I know you are all intimately aware of to frame the argument. Corrections are,
as always, welcome.
>
> To put what Alex wrote in a somewhat different way, I'd say that the
> tension here is between explicit configuration (as one finds today in,
> e.g., the Maven world) and implicit configuration (IoC).
Just a small nit: IoC can also be explicit, its just that the explicitness is
decoupled from the module, and controlled by another party, allowing for more
flexibility in the assembled system.
> Both approaches
> are important. The former is typical of standalone Java SE applications
> while the latter is typical of Java EE applications, though the two
> approaches are often intermixed.
I agree they are certainly intermixed elements of a system, but I’d also argue
IoC is pervasive in SE applications as well (e.g. inclusion of 330 and 250 in
SE are examples of a desire for SE usage). I can’t refute that it has greater
usage in EE, since its part of the spec, and thus effectively every EE
application.
I think a better use case categorization of this problem is static linkage vs
dynamic invocation. In static linkage an explicit symbol mapping resolved by
the language itself is ideal as it avoids ambiguity, and by definition is
static. On the other hand with dynamic invocation it’s common for the caller to
utilize introspection and discovery as part of the natural flow of executing a
dynamic call. Resolving ambiguity is not an issue in this case, since it is
already handled by the caller as part of introspection.
>
> What we have in the design today seems to support the explicit approach
> pretty well, but we're still trying to figure out how best to support the
> implicit approach.
(Thanks for trying to address this concern!)
>
> If I understand correctly, your view of the present proposal is that:
>
> (1) It induces too much boilerplate, requiring developers to write
> `exports dynamic P` for every single package `P` that's subject
> to reflection by a framework, and
That’s an accurate summary of this point. To the user they have already
expressed their intention by including an annotation on their class, or
expressing it in configuration. This requires that they restate that intention,
in another area. It’s also a potential source of errors/confusion if a user
unintentionally expresses an inconsistency.
>
> (2) It weakens encapsulation too much, by making the types in such a
> package available for reflection at run time by any module in the
> system.
Sorry for the confusion, what I was trying to say on this point was a bit
different. What I was trying to say was:
(2') It weakens encapsulation by forcing the introduction of exports
introducing potential conflicts that break applications.
As an example, assume I have three modules with classloader-per-module
isolation (A, B, and Victim)
- A exports foo, and has a non-exported package “bar"
- B exports bar
- Victim has a module-info with requires A; requires B
Now A decides to use IoC on some of its classes in bar, so it’s definition is
changed to:
{ exports foo; exports dynamic bar; }
Since exports dynamic is internally a normal export at runtime, module
resolution fails when loading Victim, because its now including a duplicate
package, even though A had no intention of publishing its internal bar package
for linkage.
Qualified exports could in theory address this problem, but they are
problematic in a dynamic environment since the module is simply not in a
position to know all of the various modules which would/could enhance it.
Notably, a key aspect of IoC is that modules are decoupled. For example, a
container might employ dynamic selection of multiple versions of the code
performing the enhancement, and/or it might segment its implementation into
multiple implementation modules. Additionally the code behind a module that
uses qualified exports is less reusable, and has to be updated according to
everything which may or may not reflectively access it. Ultimately you would
end up needing to dynamically generate them as you mention later on, but there
are still problems there as well (I’ll expend further below).
>
> These observations lead to your suggestion to allow declarative module
> boundaries to be overridden by "trusted" framework code. It's far from
> clear how to define such a facility in a way that would still allow us to
> achieve one of our primary goals, namely strong encapsulation, i.e., the
> ability of the author of a module to declare which types are accessible
> by other components, and which are not.
My understanding was the underlying driver was a security concern (that’s were
I was going with that suggestion). Is that accurate?
I think the goal you list above is laudable but I’m hoping there is room for
nuance.
One of the other points I was making in my earlier writeup is that you have two
different access roles at play here. There is interaction with a module through
an API/contract, and you have runtimes which enhance/augment the implementation
of a module. Encapsulation is very important for the former, but it’s
counterproductive for the latter. For example, the JDK itself needs to break
encapsulation for some of its own actions, in order to implement rich
functionality (e.g. Java serialization on non exported packages), but it also
needs to be able to do so without weakening the contract between end-user
modules. Containers and frameworks need the same ability to be able to extend
the platform with the same powerful capabilities that have already been
delivered on Java through Java 8, and to further innovate in the future.
> I'd therefore like to explain
> and explore how the above issues can be addressed with the present
> design.
>
> * * *
>
> To point (1), we all know that the most common way for developers to
> write Java code today is with a rich and powerful IDE. These tools
> already have plenty of built-in cleverness for generating POJO classes,
> deriving precise `import` directives, and ameliorating other kinds of
> boilerplate. I don't think it would be at all a stretch for such tools
> to generate precise `exports dynamic` directives on demand, based upon
> the presence of IoC-style annotations, and maintain their consistency
> over time. Just as precise `import` directives in class and interface
> declarations document dependences upon specific types, so precise
> `exports dynamic` directives in a module declaration would document
> the exposition of specific types for reflection at run time.
>
> If we think it likely that some modules will need to export dozens or
> hundreds of packages, leading to extremely long module declarations, then
> one possible refinement would be to allow a wildcard: `exports dynamic *`
> would export all of a module's packages for reflection at run time. This
> would likely be straightforward.
It’s certainly true that tooling can address, and even negate the issue.
However it does require that the IDE understand the specific framework in use.
For large standards like Java EE, that’s a reasonable expectation. Although,
there are many different frameworks and runtimes that do this, so coverage will
likely be incomplete and or/lag. I suppose an IDE could just default to
exporting everything but then that makes conflicts more likely.
>
> * * *
>
> To point (2), if some packages in a user module need to be exported for
> reflection at run time, and a container wishes to ensure that only select
> "trusted" framework modules can access the types in those packages, then
> that's already expressible today. We can also ensure that the set of
> packages exported by a module is the same whether it's used standalone
> on Java SE versus inside a container, which as you observe elsewhere in
> this thread [1] could be problematic.
>
> Suppose, e.g., we have an application module that's written against JPA,
> rather than any specific JPA implementation, and exports the package
> containing its entity classes for reflection at run time:
>
> module com.foo.data {
> requires java.persistence;
> exports dynamic com.foo.data.model;
> }
>
> When used standalone, outside of a container, this module will export the
> package containing its entity classes for reflection at run time. The
> classes will be accessible to every other module, but from a security and
> integrity standpoint we assume that whoever invokes the run-time system,
> i.e., whoever provides the command-line arguments to the `java` launcher
> or its equivalent, is trusted to ensure that no adversarial modules are
> present.
>
> When used inside a container, the container already has the power to
> prevent an adversarial module from accessing the module's entity classes.
> That's because we expect containers to load every application into a
> unique layer [2], and a container can rewrite module descriptors when
> configuring a layer.
Right, for the reasons I listed above, this is really only workable if the
container rewrites these values and/or adds them itself. However, there is
still some challenges with this. Another capability could come online in a
running system in a hot fashion, after the qualified list was computed during
initialization. So the set would need to be expanded to not just what is
required now, but all possible consumers that could be required, and these may
not be known yet. So as an example, the container administrator hot deploys a
new service which snapshots internal state. The module implementation is not
known until that code is deployed, forcing the container to bounce all
deployments just to recompute the export list.
Perhaps this problem could be addressed by addressed by an indirection that
represents a role or an actor (e.g. something like "export dynamic * to role
runtime”). I haven’t thought that through though.
> This is nothing to be ashamed of -- we fully expect
> it to become a common practice.
>
> If the container is set up to provide, e.g., Hibernate to this particular
> application, then it could narrow the accessibility of the entity classes
> by rewriting the above module declaration to refine the `exports dynamic`
> directive:
>
> module com.foo.data {
> requires java.persistence;
> exports dynamic com.foo.data.model
> to hibernate.core, hibernate.entitymanager;
> }
>
> (This is one of the very few use cases for qualified dynamic exports.)
>
> Whether standalone or in a container the same set of packages is exported
> by the module; the only difference is that, inside the container, the
> exports are qualified.
>
> * * *
>
> To sum up, for (1) I agree that unnecessary boilerplate is a bad thing.
> Asking the author of a module to be explicit about which packages are
> exported for reflection at run time, however, is of high value when
> trying to understand how the module fits into a larger system. The cost
> of such explicitness can, moreover, easily be reduced by the tools that
> almost all Java developers already use.
>
> For (2), I share your concern and I think it can be addressed within the
> scope of the present design. At this point I don't see a strong need to
> introduce a way to enable framework code to violate module boundaries
> arbitrarily at run time, and I don't know how to do that without,
> essentially, giving up on one of our primary goals.
Thanks for sharing your perspective. I can respect pushing hard on an ideal.
IMHO tweaking, or perhaps slightly reinterpreting, this goal doesn’t mean it
wasn’t achieved, it’s just adapting to accommodate a very valuable set of use
cases. Another way to look at it, is that once you have containers modifying
and generating descriptors you have already transferred authority from the
module to the runtime, so why not formalize that in a mechanism that best
enables the use case? I’m hopeful we can find a way to do so.
>
> We could make it easier to rewrite module descriptors, by providing an
> API for that purpose rather than expecting container developers to use
> libraries such as ASM, and perhaps that's worth doing, but it's a
> different issue.
It would be nice if there was a way to provide and/or alter a module definition
in something other than in bytecode, mainly for optimal generation reasons, but
I certainly understand that some things will have to wait as future
enhancements.
>
> - Mark
>
>
Thanks again
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat
