Module change detection...
There are a number of class loading optimizations (including HotSpot's
Class Data Sharing) that are hard to make generic due to the lack of a
standard way to validate cached class state. We have a unique
opportunity to fix this in 277...
I propose that we add:
1. An annotation to record a digest over the module's contents, which
would then be visible from ModuleInfo.
2. A convenience api to return the digest, e.g.:
public interface ModuleContent {
...
public byte[] getDigest();
}
The jam tool would compute and add the annotation, and the default
implementation(s) of ModuleContent would then just return the value from
the annotation.
Other implementations would be free to compute the value at install time
(or even runtime); development environments could generate a value
using a simple counter.
Unlike the version number, this value can be used to ensure that any
change to the module's contents can be detected. Both HotSpot and other
JVMs can then introduce optimizations like Class Data Sharing that
depend on being able to validate the cache file.
It may similarly be useful to have ModuleInfo be able to (lazily)
compute a digest for itself, and/or have a digest on ModuleDefinition
that encompasses both the metadata (ModuleInfo) and the contents.
// Bryan
Re: Refactorability proposal...
Yes, this could be an issue. If JavaEE:4.0.0 merely imports and re-exports the specific packages, the imports can just be copied out. If it actually contains them, and they themselves declare their own versions, the jam tool could emit imports for those specific versions, but it cannot generate ranges. This brings us back to the contract idea: a named/versioned collection of import constraints. The indirection afforded by this could certainly solve this problem. And we do have it, sort of: a module that contains no classes or resources itself, only imports and re-exports. In the current model, this is a bit of a hack because the module will actually be instantiated with its own loader, which is a waste of cycles/resources. If, OTOH, we special case such contract (indirect?) modules such that they are not instantiated (or at least don't have a loader) but act only as a bag of import constraints that can be borrowed by an importer, that seems like a reasonable solution. This could be simplified by doing the borrowing at build time, as in my proposal, at the cost of some loss of flexibility. // Bryan Adrian Brock wrote: Another issue would be how you handle the generated package constraints. e.g. Suppose you a module JavaEE:4.0.0 that contains packages javax.jms:1.1.0, javax.resource:1.5.0, etc. I then declare a module constraint JavaEE:[4.0.0, 5.0.0) i.e. I work with JavaEE4 but not JavaEE5 What do the package constraints resolve to for jms and jca if we don't have a JavaEE5 module in the repository? :-) JavaEE is not a very a good example since package versions wouldn't change across a major release, but they could in other cases or more general module version constraints. On Tue, 2008-06-10 at 18:27 -0700, Bryan Atsatt wrote: (FYI: I've discussed this with Stanley but wanted to ensure it was visible to everyone.) Proposal 1. Eliminate runtime use of import-by-module in the JAM system: a. Support the (very convenient) import-by-module at the source level. b. jam tool transforms import-by-module to a list of import-by-package statements. 2. Add APIs to fully support import-by-package in the JAM system: a. Support Version annotation in package-info.java (or in module-info.java). If a package does not declare a version, it inherits that of the enclosing module. b. Add ImportPackage annotation with version constraints. 3. Add APIs to fully support import-by-package in the abstract framework: a. Add methods to Query to produce import-by-package nodes. b. Replace Query.getIndexableNames() with fully generic variants (I proposed a solution here previously which I will re-post). Rationale Module refactoring is inevitable, particularly during the transition from the current, effectively flat class space to a fine-grained space provided by module systems. We have significant experience with this issue at Oracle (with the transition to our own module system), and OSGi best-practices for conversion include starting with everything in one bundle and then separating out pieces as experience is gained. A very common pattern, in our experience, is for developers to start with many extra jars in their initial module (a mini version of class-path hell). As that module is put into wider use, someone discovers that package X is also contained in their module, and that duplication either leads to runtime conflicts (very bad), or just plain footprint bloat. The obvious answer is to put package X in a separate module, and have everyone share it via imports. But... not so fast. If there are consumers of that module who import it by module name alone, then pulling X out of it will cause those importers to break. And if it is possible for your module to have been imported by *anyone* by name alone, then you are stuck: either you break them or you live with the incorrect granularity (which just isn't an option in the conflict scenarios). Not a happy choice. Originally, I had proposed to do away with import-by-module altogether, both to avoid this problem and to eliminate the conceptual disconnect. Your code does not today contain import statements that name *jars*, it names packages and/or specific classes in those packages. Why invent a new system that takes such a large step backwards? The answer is simply convenience. Imagine a module that contains 100 packages and it is obvious that writing a single import statement is far easier than discovering and explicitly writing all the package imports. Yes, IDEs will likely mostly eliminate this issue, but it still makes sense to be able to do this by hand. This proposal is an attempt to maintain the convenience while adding the crucial ability to safely refactor: step 1b is the central idea. // Bryan
Re: Refactorability proposal...
The proposal is to resolve module--packages at *build* time, not deploy. This way, the mapping is guaranteed to be valid. And yes, this does make the sysadmin job a bit more difficult, but that should be relatively easy to address with tooling. The main point is that relying on import-by-module at runtime guarantees that refactoring is problematic. My proposal seems like a reasonable way to deal with this, but I wanted to start this discussion to explore others as well. Any thoughts on how to solve this problem? // Bryan Adrian Brock wrote: Refactoring: Aren't there cases where resolving a module import into its packages during deployment into the repository will break with some refactorings? I'm not talking about the more stable module refactorings where modules are broken into other modules (which you seem to be addressing). This is more development time practices where somebody is refactoring module contents. i.e. the importing module could end up with a stale view of what it should be importing. A not very good example would be my module B imports a module A where A exports com.acme.a.interfaces and com.acme.a.implementation I don't use com.acme.a.implementation explicitly so it shouldn't really be one of my constraints. Somebody then refactors module B such that the implementation is now in com.acme.a.impl but my module wants to import the now non-existant (or at least stale) com.acme.a.implementation. Overrides in the repository: I know we avoid these discussions on the list since it is really a tooling issue. But it is a lot easier for a sysadmin to change a single version constraint for a module import than it is to have to figure out what package versions they need to change on the package imports and change each one individually. Basically, the convenience of import-by-module extends beyond the development/source code phase. OSGi/291 There's still going to be a kind of module import somewhere to handle OSGi's bundle import constraint. i.e. OSGi bundle B issues a request to import bundle A which is really a 277 module. On Tue, 2008-06-10 at 18:27 -0700, Bryan Atsatt wrote: (FYI: I've discussed this with Stanley but wanted to ensure it was visible to everyone.) Proposal 1. Eliminate runtime use of import-by-module in the JAM system: a. Support the (very convenient) import-by-module at the source level. b. jam tool transforms import-by-module to a list of import-by-package statements. 2. Add APIs to fully support import-by-package in the JAM system: a. Support Version annotation in package-info.java (or in module-info.java). If a package does not declare a version, it inherits that of the enclosing module. b. Add ImportPackage annotation with version constraints. 3. Add APIs to fully support import-by-package in the abstract framework: a. Add methods to Query to produce import-by-package nodes. b. Replace Query.getIndexableNames() with fully generic variants (I proposed a solution here previously which I will re-post). Rationale Module refactoring is inevitable, particularly during the transition from the current, effectively flat class space to a fine-grained space provided by module systems. We have significant experience with this issue at Oracle (with the transition to our own module system), and OSGi best-practices for conversion include starting with everything in one bundle and then separating out pieces as experience is gained. A very common pattern, in our experience, is for developers to start with many extra jars in their initial module (a mini version of class-path hell). As that module is put into wider use, someone discovers that package X is also contained in their module, and that duplication either leads to runtime conflicts (very bad), or just plain footprint bloat. The obvious answer is to put package X in a separate module, and have everyone share it via imports. But... not so fast. If there are consumers of that module who import it by module name alone, then pulling X out of it will cause those importers to break. And if it is possible for your module to have been imported by *anyone* by name alone, then you are stuck: either you break them or you live with the incorrect granularity (which just isn't an option in the conflict scenarios). Not a happy choice. Originally, I had proposed to do away with import-by-module altogether, both to avoid this problem and to eliminate the conceptual disconnect. Your code does not today contain import statements that name *jars*, it names packages and/or specific classes in those packages. Why invent a new system that takes such a large step backwards? The answer is simply convenience. Imagine a module that contains 100 packages and it is obvious that writing a single import statement is far easier than discovering and explicitly writing all the package imports. Yes, IDEs will likely mostly eliminate this issue
Re: Updates on Interoperability
Richard S. Hall wrote: Bryan Atsatt wrote: Richard S. Hall wrote: Sam Pullara wrote: Did we decide which way it will be compatible? I'm a little concerned that implementing something that can use OSGi modules is a far bigger task that implementing something that OSGi can use. If we are going for total compatibility it seems like we should just use OSGi. Personally I wanted something in Java that was simpler and cleaner that OSGi-users could leverage if they wanted. But I'm coming at this from the we-need-something-like-maven/gem/ivy camp. The vast majority of people that are going to use this system will have never heard of OSGI nor care about any of their more subtle features and just want an easy way to leverage the vast amount of libraries available (virtually all of them are not OSGi modules today but are jar files in maven repositories). We shouldn't be specing out some sort of uber container but rather a simple way to tie code to its dependencies. If this is all that people wanted, then we could just define repositories and associated metadata and forget about all runtime modularity support, but I strongly disagree that this is all people want or else there wouldn't be such an upsurge in OSGi interest and adoption. I strongly suggest KISS applies to our effort. I write this knowing that the committee seems to be heavily in favor of this alternate view that we implement the kitchen sink. I think our interest is similar, but perhaps our conclusions are different. Initially, I think my comments were construed as an argument against split packages in general, but I am really arguing the same as you (I think) that we don't really need split packages to be a concept in our interoperability API, if that is how we are to view the 277 API. Of course, I am also willing to have the former argument too, but I will leave that for another time. :-) From my point of view, I do not think it is wholly reasonable to assume that we want to try to accomplish interoperability across module systems, where that includes arbitrarily sharing split packages across module systems. Nor do I, but I'm trying to understand the implications if we assume that OSGi will continue to split packages. Let me try to accurately conjure the peculiarities of split packages in the OSGi module layer... We support split packages in two ways, via required bundles (i.e., module dependencies) and via bundle fragments. In general, we recommend that if you split your packages that you attach mandatory attributes on them so that importers do not get them by mistake. By requiring a bundle, you can ignore mandatory attributes and get everything a bundle has to offer and combine it with other required bundles that might be exporting other parts of the split packages. Given this, there is a semi-reasonable chance* that we can tell you which bundles are contributing to the complete packages, but we have no way of knowing if/when combined split packages are complete. Further, we have to assume that split package parts offered by different bundles do not overlap, because if we allowed them to overlap then we would have shadowing and ordering issues. Again we have no easy way to verify that they do not overlap, so we just assume they don't. Fragments on the other hand are a little trickier, since split packages from them are loaded from the same class loader to provide package private access (which is not afforded for split packages in the require bundle approach). The host bundle doesn't really know about fragments and it is possible for fragments to shadow or extend packages in the host bundle. Since fragments provide their split packages in an implicit way (i.e., they largely just become part of the host bundle's class path), there is no easy way to determine if a fragment is contributing to a given package or not. (* This feature of fragments also makes it so it is not really possible to know who is contributing classes to a split package in the require bundle case.) Ok, so first I assume it would make sense for an OSGi Repository to simply never return a ModuleDefinition for a fragment. I am not sure I understand your point. Host bundles would be returned and during the resolve process, fragments might be attached to that host, thus modifying its content in ways that we cannot predict. I simply meant that a fragment would never be returned by itself, as a separate ModuleDefinition. And the you don't know what you've got till it's resolved problem is going to be present in any module system for which package name is not a required component of a dependency expression. But as long as re-export is not the default behavior (it isn't), doesn't this really just boil down to the leakage problem? And this is addressed in 277 the same as it is in OSGi: you must declare such leaks as re-exports (uses). Not perfect, but probably good enough. So I think we're left
Re: Fw: JSR 291 interoperation status
I was just about to send a very similar message :^). Glyn Normington wrote: Hi Stanley Please could you provide an update on your work on interoperation between JSR 277 and JSR 291? If the work is still in progress, how's it coming along and roughly when might there be a strawman proposal to review? Thanks, Glyn - Forwarded by [EMAIL PROTECTED] on 06/09/07 04:12 PM - [EMAIL PROTECTED] wrote on 21/06/2007 09:59:02 AM: Hi Stanley Thanks for the status. Please could you indicate roughly when the strawman will be ready for initial review by this EG? Hopefully early feedback from the Expert Group would be a help... Glyn Stanley M. Ho [EMAIL PROTECTED] wrote on 20/06/2007 11:43:14 PM: Hi Glyn, That strawman is in progress. As you mentioned, it has significant challenges, so it takes more time to produce compared to other strawmans. Please stay tuned. ;-) - Stanley Glyn Normington wrote: Please could you give us an update on your work on interoperation with JSR 291 and some idea of when a strawman is likely to be available for initial review by the JSR 277 Expert Group? Like I said when we spoke at JavaOne, the approach you are planning on taking *might* work, but it has significant challenges, so I'd like to hear how it's coming along. Thanks, Glyn Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU / / /Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU/
Re: Module system notification mechanism
(Sorry, somehow this message got marked read and I only just now saw it.)
Comments inline.
Stanley M. Ho wrote:
Bryan Atsatt wrote:
Stanley, I assume you are proposing a general mechanism in which any
code can register listeners? If so, then I have to agree with Glyn that
this is not sufficient to implement a module activator.
The mechanism I have in mind can be registered only by code that have
the appropriate security permissions, and untrusted code in the sandbox
won't be able to listen to the notification.
I am still a bit puzzled by why the module's state needs to be coupled
directly to the result from the module's activator because this looks to
me more like a solution to a problem than a requirement. For that model
to work, I think there are couple underlying assumptions, but they might
not be valid in our case:
1. If a module's activator is executed before the module is fully
initialized, it can safely register something into some service lookup
mechanism in the system.
2. The service lookup mechanism will make sure the registered thing from
that module is not usable by other modules unless that module becomes
fully initialized.
3. If the module's activator fails to execute when the module is being
initialized, the module system knows how to work with the service lookup
mechanism to undo all the unregisterations automatically and correctly.
So do you expect that java.util.ServiceLoader will do this?
In this case, the module system probably needs to be highly integrated
with the service lookup mechanism, and that mechanism should be the only
one used by all the modules' activators. However, in practice, multiple
service lookup mechanisms exist and can be used in the Java platform,
and I don't think we should expect the module system will know how to
work with these arbitrary mechanisms to provide the suggested module's
state semantic in conjunction with the module's activator. Also, from
our previous discussions, I think our consensus is to avoid pushing the
service mechanism down to the module layer if possible.
I agree that we shouldn't try to push a *service* mechanism into the
module layer.
But a generic *activator* certainly seems appropriate. Some modules will
need to perform initialization prior to use (e.g. process a config file,
check some environmental constraints, populate a JNDI context, etc,
etc). Successful initialization for such a module is then a
pre-condition for use, and any failure should be treated exactly as
seriously as a resolution failure.
Yes, this *could* be done using the notification mechanism, but that
strikes me as a rather hackish approach. This use case is not limited to
container environments, so, IMO, should be made part of the spec.
And it seems like a rather small addition:
1. A simple interface that the module can implement, e.g.:
interface ModuleActivator {
public void start(Module module) throws Exception;
public void release(Module module) throws Exception;
}
2. An annotation to declare the activator impl name.
3. Module system instantiates and invokes activator (at end of
prep/validation or a new step). An exception here would put the module
into ERROR state.
Exactly what such an activator does is mostly irrelevant, though it
clearly cannot consume the thread; the usual precautions here should
suffice.
// Bryan
Re: Module isolation
Stanley M. Ho wrote: Hello Bryan, Bryan Atsatt wrote: ... But this approach doesn't really work very well. What happens when, moments after releasing a module, another application is deployed that needs the same module? It gets a different instance. And that could easily lead to ClassCastExceptions and/or LinkageErrors. How is it possible to know when it is safe to free up resources? The use case I have is that sometimes we might want to make a module temporary unavailable (e.g. turning off a plugin in the IDE), without shutting down the repository (possibly with hundreds/thousands of other modules) or uninstall the module. In this case, the container will trigger not only the release of the existing module instance (so it will have a chance to be GCed eventually), but it will also make the module definition invisible (through visibility policy) from other modules. Without the second part, it has the potential problem you described. This use case seems to presume that the IDE can/will ensure that there is only *one* consumer of the module: itself. What if a different plugin has a dependency on it, and has already been resolved? Is the intention that the IDE will be wired into the module system deeply enough to manage this correctly? Again, this is why we need either a real, general, isolation model, or an access control model: Containers, of any kind, must be able to explicitly control access to private modules. An application server, for example, will need to keep one application from accessing the modules of another. And it must *know* that there is no sharing, so that the application lifecycles can remain independent. So we need some notion of a context. A purely private repository instance is one (probably good) possibility. Another is the wrapper Repository approach, but this requires definition copies (and management of sharing content, lifecycle, etc). I started this thread with another, explicit type (ModuleContext), but it isn't obvious how to use such beast correctly. An access control model would also work, where all parties can share Repository instances, but we somehow discriminate between *callers* to return different results. We may even want to support some mixture of private + access control. Regardless of what approach we take, the releaseModule() idea is too simplistic. Having originally created the detach() method, thinking along similar lines as you are with the plugin case, I do understand the idea; I just no longer think it is sufficient :^). The only safe time to release a module is when there are *zero* importers, and, even then, you must hide/release atomically, ensuring that no new imports spring up during the operation. Note that I don't think this is a common thing many developers want to do. In fact, I think we should discourage most developers from calling releaseModule() because of the potential consequences. On the other hand, we shouldn't preclude this use case either. If you have better suggestion to address this use case, I would like to hear it. Well, I agree in theory. But... I am struggling to understand how we provide an isolation model. If: - There is a 1:1 relation for ModuleDefinition-Module instance, and - Isolation requires separate Module instances, then - Isolation requires separate ModuleDefinition instances If this is our isolation model, then how does a ModuleSystem instance support this? Clearly, it would need to keep a mapping from each ModuleDefinition to its Module instance. How is this simpler, or better? In your current model (just as in my detach() model), there is still a 1:1 from *at any given moment*, at least from the perspective of the definition. Are you thinking that the ModuleSystem would have to keep track of released modules? The ModuleSystem instance would have a ModuleDefinition, Module mapping, and this should be a very simple thing to support and maintain. Also, the ModuleSystem needs to maintain this information anyway to avoid multiple Module instances to be instantiated for a given ModuleDefinition, or to avoid a Module instance to be instantiated if a ModuleDefinition has been uninstalled or belongs to a repository which has been shutdown. And yes, the ModuleSystem would have another map to keep track of all the ModuleDefinitions that are no longer usable/instantiated. Having the information centralized in one place has other benefits too, e.g. if we want to find out what the outstanding module instances from module definitions in all repositories, the ModuleSystem can provide the answer easily. Sure. (And the ModuleSystem could not use strong references to hold released Module instances, else it would prevent GC.) My view is that ModuleSystem needs to keep track of various runtime information related to ModuleDefinition and Module anyway, so I don't see clear benefit in moving part of that information into other classes. Other than Module instances, what other runtime information
Re: Exported resources
Whoops, .jsp files are not candidates for private resources, since the
*container* must have access to them. But I stand by the gif/html/xml
comment :^).
Another candidate is property files (thanks to Stephen McConnell for
reminding me), when used as default configuration.
Bryan Atsatt wrote:
Stanley M. Ho wrote:
Hi Bryan,
Bryan Atsatt wrote:
Yes, I think we're getting close now :^)
(By privileged reflection, you mean having permission to call
AccessibleObject.setAccessible(true), right? If so, then we're back to
the grant issue!)
Yes. In this case, it's the ResourceBundle API calling it to retrieve
ListResourceBundle, so it'll have the sufficient permissions to do that.
Anyway, I think we should leave this issue aside for now.
I still think we can provide private, (i.e. non-exported) resources
which *are* available to the module classes, but to no class outside
of it.
Without using permissions.
Remember this?
private static class StackAccessor extends SecurityManager {
public Class[] getStack() {
return getClassContext();
}
}
private static final STACK_ACCESSOR = new StackAccessor();
If a resource is found, and is not-exported, a module class loader can
use this (or similar) to get the immediate caller's class (at a fixed
index).
If the caller class' Module does not match that for the loader, we don't
return the resource.
I think we should step back to rethink the use cases a bit.
Typically, a module calls some APIs (e.g. ResourceBundle, AWTToolkit,
etc.) and these APIs might then retrieve resources through the module's
ClassLoader.getResource*(). If we are saying that a module is required
to make these resources exported in order for these use cases to work,
then what are the actual use cases we want to support by hiding
resources from getResource*()?
I think there remains a common use case where hiding resources would be
a good thing if developers want to hide all the resources that are not
used by anyone through ClassLoader.getResource*() (e.g. all the .class
files). On the other hand, I think the use case where the code in the
module calls its own ClassLoader.getResource*() to retrieve private
resources directly is not very common; If this is a rare use case, I
don't think it makes sense to introduce this notion of private
resources. If we are forcing developers to export resources for the
common use cases (like ResourceBundle and ServiceLoader) anyway, then
why not require them to export resources in other cases as well?
It would help if there are some real use cases to support why it is
important to have this notion of private resources.
One very common use case comes from the EE world: web modules containing
html/gif/xml/jsp files, all of which are implementation details.
I would assume that most any app with a UI would have similar resources.
- Stanley
Re: Import specifications (was Re: Import constraints)
Bryan Atsatt wrote:
Stanley M. Ho wrote:
Hi Bryan,
Bryan Atsatt wrote:
...
By having Dependency extend Query, instances can be directly passed to
Repository.find(); no conversion required. We could obviously just make
a getter for the Query if we want, but why not make this convenient to
use?
An import dependency describes the relationship between two modules.
Agreed. And I think we would be far better off thinking of and modeling
this relationship as a general *requirement specification*, rather than
using the very narrow definition:
name + version(s)
Clearly these are important elements of an import specification. But the
design should allow for and even support others.
One concrete example is attributes. What is the point of declaring them
on a module if they cannot be used in an import specification?
Sorry, I meant to change this before sending. I do know that you meant
for attributes to be used by higher level mechanisms (e.g.
ServiceLoader). And that is certainly a valid use case (though it isn't
entirely clear to me how this works other than by polling).
But enabling them as import specifiers seems like an even more natural
use case.
And I think these could be very valuable.
The term constraint makes sense when you think of name as the primary
specification, and everything else as modifiers of it.
But this puts all the emphasis on the wrong syLLable, IMO. What will we
do when a contract model is introduced, where attributes and
version(s) are all that is required to specify the desired contract?
Module name won't be required at all in such a specification.
So far, we have three distinct actors in the resolution process:
1. Developer: creates import specifications, using whatever selection
criteria makes sense.
2. Admin: creates import specification *filter* (ImportOverridePolicy),
mapping developer specification to fit the local environment. (Can also
use a VisibilityPolicy, but it isn't clear to me why that same
functionality can't be achieved in the override policy.)
3. Module System: locates definitions that match specifications.
All of these actors collaborate to produce a single list of candidate
definitions at runtime. If that list contains duplicate packages, the
module system must select the best fit, or fail.
Clearly the Query mechanism provides an extensible solution to the
problem in #3. And annotations give us an extensible solution to the
problem in #1.
But, just as clearly, we have a big gap in the middle: the filter model
is not a general, extensible mechanism. And I think we need to fix this.
It seems to me that the best solution is to use the *same* model to
filter as we use for lookup: Query.
(Please see the Query optimization thread for my proposal to change
Query from an opaque to a transparent type.)
So, rather than pass VersionConstraint instances through the narrow()
method, we can pass Query instances. Yes, this does make it a bit more
work to implement such a filter, but it is then a completely extensible,
symmetric model.
And this is also why I proposed that we change ImportDependency to
simply take a Query as a ctor argument:
public ImportDependency (Query spec,
boolean optional,
boolean reExport) {...}
(I should have sent my optimization proposal before I suggested the
above change, so it would've made more sense. Really this would be so
much easier if you could just read my mind :^)
Further, I don't understand the need to limit the filter to only
narrowing operations. For example, why shouldn't an admin be able to
map a module *name* as well? This could be really handy (and might even
allow for elimination of the refactoring/name problem).
In fact, why not allow the admin to be able to filter *any* part of the
import specification? I would prefer to see a more generic model than
the current ImportOverridePolicy, for example:
public class ImportOverridePolicy {
public ListImportDependency getImports(ModuleDefinition def) {
return def.getImportDependencies();
}
public static ImportOverridePolicy getPolicy() {...}
static void setPolicy(ImportOverridePolicy policy) {...}
}
The default implementation does nothing, but subtypes can perform any
transformation they want. The ModuleSystem just calls this method
instead of calling the definition directly.
It is true that we may want to query a module described in the import
dependency, but an import dependency itself is not a query. In fact,
someone may even want to query a module which contains the specific
import dependency, although I think this use case is very rare. Anyway,
it sounded like the main benefit to have import dependency extend query
is to make it easier to pass into Repository.find(). Unfortunately, once
a class is extended from another, all the public/protected
methods/fields from the parent class would automatically become part of
the signature of the child class.
No kidding? :^)
I don't
Re: Relationship to JSR 291 [was: Re: Bryan's comments]
[snip] Richard S.Hall wrote: My question below still stands. In general, it seems to me that resolution consists of: 1. Selection of candidate module/bundles based on import declarations. 2. Class space consistency validation (which may narrow the list if multiple choices, or result in a failure). I don't understand what possible difference it could make whether we use a module name or a package name to select candidates in step 1. Or even if some custom code makes that selection (though I'm not a fan of this either.) In general, I can say your two steps above are correct...Felix actually has what amounts to a two-pass resolve that does (1) from above in the first step and (2) from above in the second step. However, it turns out that (2) is actually quite difficult and difficult to do in an efficient way. Perhaps it is more difficult for the OSGi framework due to some of the sophistication of its model, e.g., package substitutability, fragments, split packages, package-level uses constraints, etc. All I know is that we are still finding cases that need to be clarified in the OSGi framework for this process... I won't say that I am disagreeing with you...if we can strip away some of the complexity of the OSGi model, then I am sure you could implement a reasonably straightforward resolver. Right, that is what I'm trying to get at here. I believe *very* strongly that import-by-name is a bad idea. Period. It is just plain wrong to do the equivalent of import foo.jar. We can do this already, with manifest Class-Path. Simply layering version support on it does *not* eliminate jar-hell! It is far too brittle, raising all kinds of maintenance issues. Under import-by-name, refactoring *requires* introducing a facade module to avoid breaking importers. And this facade module now becomes *another* product artifact to be managed: documented, maintained, distributed, deprecated, and ultimately perhaps removed. I truly believe that the Java world will be far better served by having 277 support ONLY import-by-package! But there has been this background fear that import-by-package is somehow terribly complex. I can certainly believe that the union of all the OSGi features *does* result in a lot of complexity. I just think that is almost entirely orthogonal to *this* domain, since we are not trying to expose all those other features. That won't necessarily help us in achieving interoperability, though. Absolutely; regardless of the outcome of the import-by-package issue, we still need to deal with exactly how the loaders can share classes and resources... // Bryan
Re: JSR 277 EG observer mailing list
How are people supposed to find this? Shouldn't there be an easily spotted link somewhere on the public 277 page? // Bryan Stanley M. Ho wrote: Hi JSR 277 experts, Just an update. This EG mailing list has been hooked up with the observer list over the weekend, and you will find the public mailing list archive here: http://mail.openjdk.java.net/pipermail/jsr277-eg-observer/ - Stanley
Re: Exported resources
I don't think I'm missing anything, just looking at it from perhaps a
different perspective :^)
First, 294 will determine accessibility for non-exported classes. The
assumption at the moment is that they will not be accessible to *any*
class outside of the module.
I do understand the value of a friend semantic; it would certainly be
nice to grant certain frameworks special access. But, so far, that is
not on the table for classes.
And if we don't have it for classes, I can't see why we should have it
for resources.
// Bryan
Glyn Normington wrote:
Some feedback from an observer which may help:
Bryan misses the need for extender model to load internal impl
classes ala
Bundle.loadClass in OSGi.
You do not want to have to export the service impl class from a module.
You want to hide the class from others' casual loading. However an
extender bundle (e.g. ServiceLoader) will need to be able to load that
class to make it instances of it available to others under the service
interface class.
Glyn
*Bryan Atsatt [EMAIL PROTECTED]* wrote on 30/05/2007 21:21:36:
I've been assuming that Module private resources should not be visible
to *any* class outside of the module. Including ResourceBundle, or any
other existing framework classes that do resource lookups (e.g.
ServiceLoader, JSF, etc). If resources need to be visible to these
existing classes, they must be exported. The very simple check I
proposed (immediate caller) is sufficient to make this assertion.
I believe your point is that if we used the permission model instead, it
would become possible for a module to invoke an external class (e.g.
ResourceBundle.getBundle()) and enable *it* to successfully load a
private resource from the module.
Aside from the permission *grant* mechanism this model would rely on, it
is an entirely different model than that used for classes! (Though we
haven't explicitly defined this in 294, it seems extremely unlikely that
we will rely on permissions--none of the other access modes do so.) Such
asymmetry is very disconcerting to me, and, I believe, just plain
wrong...
Consider that you could grant the ServiceLoader, for example, access to
a resource that names a class that it could not instantiate. That class
would have to be exported. I believe the resource should be as well.
// Bryan
Stanley M. Ho wrote:
Hi Bryan,
Those resource-related methods in ClassLoader can be called by anyone,
including code that is part of the module, code that is from other
modules, or code that is part of the platform libraries (e.g.
ResourceBundle). The approach you described would require walking the
stack to get the caller's Module, but the real issue is that it is
difficult to determine who the actual caller is from the stack.
Treating the immediate caller on the stack as the actual caller
wouldn't
be sufficient because the immediate caller could be called by someone
else who is the one actually making the call. On the other hand,
treating the originated caller on the stack as the actual caller would
be the right semantic, but this is basically the same as the security
permission approach.
- Stanley
Bryan Atsatt wrote:
Both solutions require stack walking (unless there is some new
implementation of the java security model I've not yet seen!).
The permission check does much more work than is necessary here.
Take a
look at AccessController.checkPermission() to see what I mean.
And actually there is a very simple API to get the stack, which I've
used for years:
private static class StackAccessor extends SecurityManager {
public Class[] getStack() {
return getClassContext();
}
}
private static final STACK_ACCESSOR = new StackAccessor();
Now the enclosing class can simply call STACK_ACCESSOR.getStack().
// Bryan
Stanley M. Ho wrote:
Hi Bryan,
Bryan Atsatt wrote:
1. Definitely agree that resource search order should be
identical to
class search order.
Glad to hear!
2. Using permissions to limit access to private resources seems like
overkill to me. The prototype implemented this in a very simple
fashion:
a. If resource is exported, return it, else
a. Get the caller's Module (get class from stack, get module
from it)
b. If callerModule == this, return resource, else return null.
The issue is that this approach still requires stack walking and
there
is no public API in the SE platform that let you implement this.
If stack walking is required for the check anyway, I think the
security
permission approach is better that it is implementable with the
existing
API in the SE platform.
- Stanley
/
/
/Unless stated otherwise above:
IBM United Kingdom Limited - Registered
Re: Relationship to JSR 291 [was: Re: Bryan's comments]
Responses inline, and a few clarifications here (I was a bit tired when
I finished this last night :^)...
The main point I was trying to make is that resolution must occur within
a specific context, but I don't think my example APIs showed that well.
I was assuming that ImportResolver had a ModuleContext as a field, but
we can make this much cleaner and easier to understand by passing it as
an argument:
public abstract class ImportResolver {
public abstract Module resolve(ModuleDefinition def,
ModuleContext ctx);
}
And we can really tie this together by adding a convenience method to
ModuleContext:
public abstract class ModuleContext {
...
public Module resolve(ModuleDefinition def) {
return getImportResolver().resolve(def, this);
}
}
Now resolution becomes simply:
context.resolve(definition);
(I also left out any use of the ImportPolicy, as it isn't yet clear to
me that it should remain a separate type in this model.)
// Bryan
Glyn Normington wrote:
*Bryan Atsatt [EMAIL PROTECTED]* wrote on 30/05/2007 07:57:59:
Andy Piper wrote:
At 23:19 25/05/2007, Stanley M. Ho wrote:
Anyway, it seems the EG consensus so far is to not add import package
support. If any EG member disagrees, please speak up.
Well, it depends on what the solution for enabling interoperation
with JSR 291 is.
Our requirement is that there must be a solution, if that requires
import package, so be it. If not then not.
Exactly.
I think we can all agree that, at minimum, interoperation means that
classes and resources are sharable *across* ModuleSystems at runtime.
Which implies that *import dependencies must be resolvable across
multiple ModuleSystem instances*. (BTW, I think we should change the
state name PREPARING to RESOLVING in 7.2.1.)
Agreed. We must avoid the trap of thinking that module system interop.
can be achieved by exposing class loaders (as loadClass will happily
load unexported classes).
So the open issue is the richness of the import language: must we
support only lowest-common-denominator, or can we do better without
over-complicating the design?
I for one would like to be able to have a single module express
dependencies on modules from both the same and different ModuleSystems,
*using the standard semantics of each*. This may be reaching too far,
but we should at least explore it seriously while we figure out what
interop means here...
At this point, I feel that is likely to be reaching too far, but I'm
happy to play along and see what we can learn along the way.
BASICS
So far, we only know of two different import semantics: module-name, and
package-name. For discussion, let's call these:
a. import-module
b. import-package
So, to start, we could:
1. Support declaration of both import types. If 294 supports imports at
all, it should be relatively easy to support both, since a superpackage
name is a module name, and it contains member package names. (Compiler
support is clearly the critical issue here, but it will obviously
require use of the 277 runtime, so the import *type* should be
transparent to it.) At worst, we'd need two new annotation types.
A superpackage name is a deployment module name in the JSR 277 model of
one superpackage per deployment module, but I don't see any reason why a
JSR 291 deployment module should not contain more than one superpackage.
So if 294 were to support import, then its import-module would really be
a superpackage import rather than a development module import.
If we end up with nested superpackages, might it make sense to model
multiple superpackages by enclosing them in a single top-level one?
2. Provide API for both import types (e.g. ImportDependency has
getModuleName() and getPackageName() methods, one of which will return
null on a given instance).
However, we know these are necessary but not sufficient. Leaving aside
the resolution issue for a moment, support for import-package also
suggests that we:
3. Enable a single module to declare different versions for each of its
member packages.
4. Enable efficient Repository lookup by package name.
I think these are relatively easy (but *could* be considered optional).
We also need:
5. Standard Query types for lookup by module and package name.
EXISTING DEPENDENCY RESOLUTION MODEL
The more interesting issue is dependency resolution. But this hasn't
been discussed in any real detail, so lets do so before talking further
about import-package. To simplify this discussion, I'm ignoring
bundled/custom import policies for now...
Resolution in the current spec is delegated to the associated
ModuleSystem instance (7.2.2 #8). While the details are not spelled out,
the expectation appears to be that
ModuleSystem.getModule(ModuleDefinition) must:
- Select an initial repository. Call
Re: Exported resources
I've been assuming that Module private resources should not be visible
to *any* class outside of the module. Including ResourceBundle, or any
other existing framework classes that do resource lookups (e.g.
ServiceLoader, JSF, etc). If resources need to be visible to these
existing classes, they must be exported. The very simple check I
proposed (immediate caller) is sufficient to make this assertion.
I believe your point is that if we used the permission model instead, it
would become possible for a module to invoke an external class (e.g.
ResourceBundle.getBundle()) and enable *it* to successfully load a
private resource from the module.
Aside from the permission *grant* mechanism this model would rely on, it
is an entirely different model than that used for classes! (Though we
haven't explicitly defined this in 294, it seems extremely unlikely that
we will rely on permissions--none of the other access modes do so.) Such
asymmetry is very disconcerting to me, and, I believe, just plain wrong...
Consider that you could grant the ServiceLoader, for example, access to
a resource that names a class that it could not instantiate. That class
would have to be exported. I believe the resource should be as well.
// Bryan
Stanley M. Ho wrote:
Hi Bryan,
Those resource-related methods in ClassLoader can be called by anyone,
including code that is part of the module, code that is from other
modules, or code that is part of the platform libraries (e.g.
ResourceBundle). The approach you described would require walking the
stack to get the caller's Module, but the real issue is that it is
difficult to determine who the actual caller is from the stack.
Treating the immediate caller on the stack as the actual caller wouldn't
be sufficient because the immediate caller could be called by someone
else who is the one actually making the call. On the other hand,
treating the originated caller on the stack as the actual caller would
be the right semantic, but this is basically the same as the security
permission approach.
- Stanley
Bryan Atsatt wrote:
Both solutions require stack walking (unless there is some new
implementation of the java security model I've not yet seen!).
The permission check does much more work than is necessary here. Take a
look at AccessController.checkPermission() to see what I mean.
And actually there is a very simple API to get the stack, which I've
used for years:
private static class StackAccessor extends SecurityManager {
public Class[] getStack() {
return getClassContext();
}
}
private static final STACK_ACCESSOR = new StackAccessor();
Now the enclosing class can simply call STACK_ACCESSOR.getStack().
// Bryan
Stanley M. Ho wrote:
Hi Bryan,
Bryan Atsatt wrote:
1. Definitely agree that resource search order should be identical to
class search order.
Glad to hear!
2. Using permissions to limit access to private resources seems like
overkill to me. The prototype implemented this in a very simple
fashion:
a. If resource is exported, return it, else
a. Get the caller's Module (get class from stack, get module from it)
b. If callerModule == this, return resource, else return null.
The issue is that this approach still requires stack walking and there
is no public API in the SE platform that let you implement this.
If stack walking is required for the check anyway, I think the security
permission approach is better that it is implementable with the existing
API in the SE platform.
- Stanley
Re: Exported resources
Hey Stanley, Sorry to be generating so much traffic while you're traveling! I'm not in any rush here, so feel free to take your time responding... Stanley M. Ho wrote: Hi Bryan, A module can use the ResourceBundle API to retrieve resources from other resource modules, but it can also use the ResourceBundle API to retrieve resources from the (target) module itself. In the latter case, the resources are currently not required to be exported from the target module. I don't think we want to force a module to export its own private resources simply because it wants to use the ResourceBundle API in this case, would you agree? No, I don't agree. That's what I said before :^). I *really* don't like the idea that I'm going to have to explicitly grant permission to some module to access my resource. This *significantly* complicates deployment. How will the grant occur, especially given that each environment can use an entirely different permission storage mechanism? Or are you thinking that the loaders in the module system will construct ProtectionDomains pre-wired with permissions? If so, exactly which entities will be blessed? Just the JRE code? What about all the existing non-JRE frameworks in use? Regarding there are differences in the access models between classes and resources, I also prefer symmetry if possible. However, the access model for classes and resources have always been different, and there are built-in JVM support for classes while there is none for resources, so it is unclear if we can completely eliminate this asymmetry after all. Sure, resources have never had JVM access control applied to them. But I don't think we should go to the other extreme (permissions) just so that I can give friend access to private resources. I'd much prefer that such resources simply be exported. And I don't see this as a particular problem, either. We've gotten along just fine without private resources so far (or maybe there is some big demand for this that I'm missing?). And what about the mismatch issue I brought up? Lots of existing frameworks use the services pattern, in which: 1. A well known resource name is used in a getResource() call. 2. The resource contains a string naming a provider class. 3. The provider class is loaded using Class.forName(). This is a simple but very powerful pattern, but it requires that *both* the resource and the named class be accessible to the framework. If we require a permission grant for the resource, but not for the class, it will be very easy to get access to one and not the other. Is it really worth opening up this potential headache to enable a friend model for resources? // Bryan - Stanley Bryan Atsatt wrote: I've been assuming that Module private resources should not be visible to *any* class outside of the module. Including ResourceBundle, or any other existing framework classes that do resource lookups (e.g. ServiceLoader, JSF, etc). If resources need to be visible to these existing classes, they must be exported. The very simple check I proposed (immediate caller) is sufficient to make this assertion. I believe your point is that if we used the permission model instead, it would become possible for a module to invoke an external class (e.g. ResourceBundle.getBundle()) and enable *it* to successfully load a private resource from the module. Aside from the permission *grant* mechanism this model would rely on, it is an entirely different model than that used for classes! (Though we haven't explicitly defined this in 294, it seems extremely unlikely that we will rely on permissions--none of the other access modes do so.) Such asymmetry is very disconcerting to me, and, I believe, just plain wrong... Consider that you could grant the ServiceLoader, for example, access to a resource that names a class that it could not instantiate. That class would have to be exported. I believe the resource should be as well. // Bryan
