Re: Updates on Interoperability
Bryan Atsatt wrote: Richard S. Hall wrote: Bryan Atsatt wrote: Richard S. Hall wrote: Sam Pullara wrote: Did we decide which way it will be compatible? I'm a little concerned that implementing something that can use OSGi modules is a far bigger task that implementing something that OSGi can use. If we are going for total compatibility it seems like we should just use OSGi. Personally I wanted something in Java that was simpler and cleaner that OSGi-users could leverage if they wanted. But I'm coming at this from the we-need-something-like-maven/gem/ivy camp. The vast majority of people that are going to use this system will have never heard of OSGI nor care about any of their more subtle features and just want an easy way to leverage the vast amount of libraries available (virtually all of them are not OSGi modules today but are jar files in maven repositories). We shouldn't be specing out some sort of uber container but rather a simple way to tie code to its dependencies. If this is all that people wanted, then we could just define repositories and associated metadata and forget about all runtime modularity support, but I strongly disagree that this is all people want or else there wouldn't be such an upsurge in OSGi interest and adoption. I strongly suggest KISS applies to our effort. I write this knowing that the committee seems to be heavily in favor of this alternate view that we implement the kitchen sink. I think our interest is similar, but perhaps our conclusions are different. Initially, I think my comments were construed as an argument against split packages in general, but I am really arguing the same as you (I think) that we don't really need split packages to be a concept in our "interoperability API", if that is how we are to view the 277 API. Of course, I am also willing to have the former argument too, but I will leave that for another time. :-) From my point of view, I do not think it is wholly reasonable to assume that we want to try to accomplish interoperability across module systems, where that includes arbitrarily sharing split packages across module systems. Nor do I, but I'm trying to understand the implications if we assume that OSGi will continue to split packages. Let me try to accurately conjure the peculiarities of split packages in the OSGi module layer... We support split packages in two ways, via required bundles (i.e., module dependencies) and via bundle fragments. In general, we recommend that if you split your packages that you attach mandatory attributes on them so that importers do not get them by mistake. By requiring a bundle, you can ignore mandatory attributes and get everything a bundle has to offer and combine it with other required bundles that might be exporting other parts of the split packages. Given this, there is a semi-reasonable chance* that we can tell you which bundles are contributing to the "complete" packages, but we have no way of knowing if/when combined split packages are complete. Further, we have to assume that split package parts offered by different bundles do not overlap, because if we allowed them to overlap then we would have shadowing and ordering issues. Again we have no easy way to verify that they do not overlap, so we just assume they don't. Fragments on the other hand are a little trickier, since split packages from them are loaded from the same class loader to provide package private access (which is not afforded for split packages in the require bundle approach). The "host" bundle doesn't really know about fragments and it is possible for fragments to shadow or extend packages in the "host" bundle. Since fragments provide their split packages in an implicit way (i.e., they largely just become part of the host bundle's class path), there is no easy way to determine if a fragment is contributing to a given package or not. (* This feature of fragments also makes it so it is not really possible to know who is contributing classes to a split package in the require bundle case.) Ok, so first I assume it would make sense for an OSGi Repository to simply never return a ModuleDefinition for a fragment. I am not sure I understand your point. Host bundles would be returned and during the resolve process, fragments might be attached to that host, thus modifying its content in ways that we cannot predict. I simply meant that a fragment would never be returned by itself, as a separate ModuleDefinition. And the "you don't know what you've got till it's resolved" problem is going to be present in any module system for which package name is not a required component of a dependency expression. But as long as re-export is not the default behavior (it isn't), doesn't this really just boil down to the leakage problem? And this is addressed
Re: Updates on Interoperability
Bryan Atsatt wrote:
Richard S. Hall wrote:
Sam Pullara wrote:
Did we decide which way it will be compatible? I'm a little
concerned that implementing something that can use OSGi modules is a
far bigger task that implementing something that OSGi can use. If
we are going for total compatibility it seems like we should just
use OSGi. Personally I wanted something in Java that was simpler
and cleaner that OSGi-users could leverage if they wanted. But I'm
coming at this from the we-need-something-like-maven/gem/ivy camp.
The vast majority of people that are going to use this system will
have never heard of OSGI nor care about any of their more subtle
features and just want an easy way to leverage the vast amount of
libraries available (virtually all of them are not OSGi modules
today but are jar files in maven repositories). We shouldn't be
specing out some sort of uber container but rather a simple way to
tie code to its dependencies.
If this is all that people wanted, then we could just define
repositories and associated metadata and forget about all runtime
modularity support, but I strongly disagree that this is all people
want or else there wouldn't be such an upsurge in OSGi interest and
adoption.
I strongly suggest KISS applies to our effort. I write this knowing
that the committee seems to be heavily in favor of this alternate
view that we implement the kitchen sink.
I think our interest is similar, but perhaps our conclusions are
different. Initially, I think my comments were construed as an
argument against split packages in general, but I am really arguing
the same as you (I think) that we don't really need split packages to
be a concept in our "interoperability API", if that is how we are to
view the 277 API. Of course, I am also willing to have the former
argument too, but I will leave that for another time. :-)
From my point of view, I do not think it is wholly reasonable to
assume that we want to try to accomplish interoperability across
module systems, where that includes arbitrarily sharing split
packages across module systems.
Nor do I, but I'm trying to understand the implications if we assume
that OSGi will continue to split packages.
Let me try to accurately conjure the peculiarities of split packages
in the OSGi module layer...
We support split packages in two ways, via required bundles (i.e.,
module dependencies) and via bundle fragments. In general, we
recommend that if you split your packages that you attach mandatory
attributes on them so that importers do not get them by mistake. By
requiring a bundle, you can ignore mandatory attributes and get
everything a bundle has to offer and combine it with other required
bundles that might be exporting other parts of the split packages.
Given this, there is a semi-reasonable chance* that we can tell you
which bundles are contributing to the "complete" packages, but we
have no way of knowing if/when combined split packages are complete.
Further, we have to assume that split package parts offered by
different bundles do not overlap, because if we allowed them to
overlap then we would have shadowing and ordering issues. Again we
have no easy way to verify that they do not overlap, so we just
assume they don't.
Fragments on the other hand are a little trickier, since split
packages from them are loaded from the same class loader to provide
package private access (which is not afforded for split packages in
the require bundle approach). The "host" bundle doesn't really know
about fragments and it is possible for fragments to shadow or extend
packages in the "host" bundle. Since fragments provide their split
packages in an implicit way (i.e., they largely just become part of
the host bundle's class path), there is no easy way to determine if a
fragment is contributing to a given package or not. (* This feature
of fragments also makes it so it is not really possible to know who
is contributing classes to a split package in the require bundle case.)
Ok, so first I assume it would make sense for an OSGi Repository to
simply never return a ModuleDefinition for a fragment.
I am not sure I understand your point. Host bundles would be returned
and during the resolve process, fragments might be attached to that
host, thus modifying its content in ways that we cannot predict.
And the "you don't know what you've got till it's resolved" problem is
going to be present in any module system for which package name is not
a required component of a dependency expression. But as long as
re-export is not the default behavior (it isn't), doesn't this really
just boil down to the leakage problem? And this is addressed in 277
the same as it is in OSGi: you must declare such "leaks" as re-exports
("uses"). Not perfect, but probably good enou
Re: Updates on Interoperability
Sam Pullara wrote: On Apr 25, 2008, at 5:19 PM, Richard S. Hall wrote: Sam Pullara wrote: On Apr 25, 2008, at 4:46 PM, Richard S. Hall wrote: Sam Pullara wrote: The vast majority of people that are going to use this system will have never heard of OSGI nor care about any of their more subtle features and just want an easy way to leverage the vast amount of libraries available (virtually all of them are not OSGi modules today but are jar files in maven repositories). We shouldn't be specing out some sort of uber container but rather a simple way to tie code to its dependencies. If this is all that people wanted, then we could just define repositories and associated metadata and forget about all runtime modularity support, but I strongly disagree that this is all people want or else there wouldn't be such an upsurge in OSGi interest and adoption. I guess that is what I am arguing for. I don't see the critical need, outside the internals of application servers and a very few plugin based tools, for runtime modularity support. And if OSGi can load a JSR-277 module then I feel like the interoperability job is done. I can understand your point of view, but I think you need to take a look around and see what kind of systems out there are creating either their own plugin mechanism or using an existing framework (like OSGi) to create a plugin mechanism. This is a huge area. It is actually one of the biggest impacts that Java made with its class loader approach, it brought code loading to the masses. I could give countless examples that fall outside the "few tools" above (heck, today I just ran across a gaming platform using OSGi), but I won't go into that. I have no problem with them using OSGi and I think it has a lot of value. It just doesn't seem like you need to make it built into Java but rather another container specification. I might wholeheartedly accept your view if the powers to be decided to pare down 277 to be just a repository model for Java JAR files, because that could potentially be fairly easy for us to specify and get working as an OSGi module repository (which is currently lacking). I have said before that it makes sense to have 294 give us needed VM support for modules, 277 give us repository support for modules, and 291 give us run-time support for modules. This seems like a good plan. We'd have no problem wrapping this up quickly with this as the goal. However, that isn't the way we have been going, nor has it ever been as far as I am aware. I agree. I've been trying to push us this way since the beginning. Unfortunately, me: FAIL. Join the club... -> richard Sam
Re: Updates on Interoperability
Sam Pullara wrote: On Apr 25, 2008, at 4:46 PM, Richard S. Hall wrote: Sam Pullara wrote: The vast majority of people that are going to use this system will have never heard of OSGI nor care about any of their more subtle features and just want an easy way to leverage the vast amount of libraries available (virtually all of them are not OSGi modules today but are jar files in maven repositories). We shouldn't be specing out some sort of uber container but rather a simple way to tie code to its dependencies. If this is all that people wanted, then we could just define repositories and associated metadata and forget about all runtime modularity support, but I strongly disagree that this is all people want or else there wouldn't be such an upsurge in OSGi interest and adoption. I guess that is what I am arguing for. I don't see the critical need, outside the internals of application servers and a very few plugin based tools, for runtime modularity support. And if OSGi can load a JSR-277 module then I feel like the interoperability job is done. I can understand your point of view, but I think you need to take a look around and see what kind of systems out there are creating either their own plugin mechanism or using an existing framework (like OSGi) to create a plugin mechanism. This is a huge area. It is actually one of the biggest impacts that Java made with its class loader approach, it brought code loading to the masses. I could give countless examples that fall outside the "few tools" above (heck, today I just ran across a gaming platform using OSGi), but I won't go into that. I might wholeheartedly accept your view if the powers to be decided to pare down 277 to be just a repository model for Java JAR files, because that could potentially be fairly easy for us to specify and get working as an OSGi module repository (which is currently lacking). I have said before that it makes sense to have 294 give us needed VM support for modules, 277 give us repository support for modules, and 291 give us run-time support for modules. However, that isn't the way we have been going, nor has it ever been as far as I am aware. -> richard Sam
Re: Updates on Interoperability
Sam Pullara wrote: Did we decide which way it will be compatible? I'm a little concerned that implementing something that can use OSGi modules is a far bigger task that implementing something that OSGi can use. If we are going for total compatibility it seems like we should just use OSGi. Personally I wanted something in Java that was simpler and cleaner that OSGi-users could leverage if they wanted. But I'm coming at this from the we-need-something-like-maven/gem/ivy camp. The vast majority of people that are going to use this system will have never heard of OSGI nor care about any of their more subtle features and just want an easy way to leverage the vast amount of libraries available (virtually all of them are not OSGi modules today but are jar files in maven repositories). We shouldn't be specing out some sort of uber container but rather a simple way to tie code to its dependencies. If this is all that people wanted, then we could just define repositories and associated metadata and forget about all runtime modularity support, but I strongly disagree that this is all people want or else there wouldn't be such an upsurge in OSGi interest and adoption. I strongly suggest KISS applies to our effort. I write this knowing that the committee seems to be heavily in favor of this alternate view that we implement the kitchen sink. I think our interest is similar, but perhaps our conclusions are different. Initially, I think my comments were construed as an argument against split packages in general, but I am really arguing the same as you (I think) that we don't really need split packages to be a concept in our "interoperability API", if that is how we are to view the 277 API. Of course, I am also willing to have the former argument too, but I will leave that for another time. :-) From my point of view, I do not think it is wholly reasonable to assume that we want to try to accomplish interoperability across module systems, where that includes arbitrarily sharing split packages across module systems. Let me try to accurately conjure the peculiarities of split packages in the OSGi module layer... We support split packages in two ways, via required bundles (i.e., module dependencies) and via bundle fragments. In general, we recommend that if you split your packages that you attach mandatory attributes on them so that importers do not get them by mistake. By requiring a bundle, you can ignore mandatory attributes and get everything a bundle has to offer and combine it with other required bundles that might be exporting other parts of the split packages. Given this, there is a semi-reasonable chance* that we can tell you which bundles are contributing to the "complete" packages, but we have no way of knowing if/when combined split packages are complete. Further, we have to assume that split package parts offered by different bundles do not overlap, because if we allowed them to overlap then we would have shadowing and ordering issues. Again we have no easy way to verify that they do not overlap, so we just assume they don't. Fragments on the other hand are a little trickier, since split packages from them are loaded from the same class loader to provide package private access (which is not afforded for split packages in the require bundle approach). The "host" bundle doesn't really know about fragments and it is possible for fragments to shadow or extend packages in the "host" bundle. Since fragments provide their split packages in an implicit way (i.e., they largely just become part of the host bundle's class path), there is no easy way to determine if a fragment is contributing to a given package or not. (* This feature of fragments also makes it so it is not really possible to know who is contributing classes to a split package in the require bundle case.) I think that is fairly complete description of the situation, sorry if it is somewhat terse. In summary, I wouldn't assume that we can get reasonable interoperability at this level. If you need these types of features, then you should stick to a single modularity framework...preferably OSGi. ;-) But my overall point is, I believe that if we can support interoperability at module- and package-level dependencies across module systems, then I think we would hit most use cases. -> richard Sam On Apr 25, 2008, at 1:57 PM, Richard S. Hall wrote: Bryan Atsatt wrote: Richard S. Hall wrote: Gordon Hirsch wrote: 1. Map its dependency declarations into a standard runtime representation. 2. Support a standard search model over its stored modules, using the runtime dependency expressions. 3. Map its stored module data into a standard runtime representation that can be returned by the search. One challenge lies in defining the "standard runtime representations" and "sta
Re: Updates on Interoperability
Gordon Hirsch wrote: Bryan Atsatt wrote: On the split package front, however, it seems a little fuzzy to me. If an OSGi implementation of 277 is also going to remain OSGi Rx compliant, it will still need to support split packages, right? Yes, and, like it or not, split packages are common enough that it may be hard to completely ignore them wrt to interoperability. If so, don't we need to surface this fact at the 277 level? Perhaps that is as simple as acknowledging that Module.getImportedModules() may return more than one instance that exports a given package. This is the case I was thinking of. Also, IIRC, OSGi allows the specification of additional attributes on Import-Package statements which can be used to help disambiguate the import in the presence of split packages. Does this facility get surfaced to the standard search mechanism? If features at this level are involved, this is where I begin to become concerned about the complexity of our task :-). Well, the attributes really have nothing to do with split packages, although the recommendation is to use mandatory attributes on split packages to avoid importers from accidentally getting wired to a partial package. We can go down this path, but I doubt it will be as simple as just changing getImportedModules() to return an array... -> richard
Re: Updates on Interoperability
Bryan Atsatt wrote: Richard S. Hall wrote: Gordon Hirsch wrote: 1. Map its dependency declarations into a standard runtime representation. 2. Support a standard search model over its stored modules, using the runtime dependency expressions. 3. Map its stored module data into a standard runtime representation that can be returned by the search. One challenge lies in defining the "standard runtime representations" and "standard search model" in a universal enough way to encompass OSGi and other module systems. This implies embracing concepts (in these standard representations and search model) that were not universally liked by the EG early on. (Split packages and package-level import/export come to mind.) Package-level import/export seem like a must, but I still don't see split packages as a necessity. Strongly agreed on import-by-package, and that ModuleDefinition requires a method to enumerate exported packages (and probably an exportsPackage(String packageName) method to enable an efficient Query). On the split package front, however, it seems a little fuzzy to me. If an OSGi implementation of 277 is also going to remain OSGi Rx compliant, it will still need to support split packages, right? If so, don't we need to surface this fact at the 277 level? Perhaps that is as simple as acknowledging that Module.getImportedModules() may return more than one instance that exports a given package. That all depends on if the 277 API is a subset of OSGi functionality or equal to it, I suppose. -> richard // Bryan
Re: Updates on Interoperability
Gordon Hirsch wrote: 1. Map its dependency declarations into a standard runtime representation. 2. Support a standard search model over its stored modules, using the runtime dependency expressions. 3. Map its stored module data into a standard runtime representation that can be returned by the search. One challenge lies in defining the "standard runtime representations" and "standard search model" in a universal enough way to encompass OSGi and other module systems. This implies embracing concepts (in these standard representations and search model) that were not universally liked by the EG early on. (Split packages and package-level import/export come to mind.) Package-level import/export seem like a must, but I still don't see split packages as a necessity. -> richard
Re: EG revitalization
+3 -> richard Bryan Atsatt wrote: This EG has stagnated and needs to be brought back to life. I'd like to propose these concrete steps: 1. Add Peter Kriens to the EG. Peter has clearly demonstrated great expertise in this area, and I believe he would quickly become a valued, active contributor. 2. Bring the interop design/discussion/investigation process out of Sun offices and into the EG. 3. Give the EG read access to the tip of the source tree (or at least very current snapshots). // Bryan
Re: Spec re-organization/update proposal...
Sounds good to me. -> richard Bryan Atsatt wrote: I propose that we re-organize and update the next draft of the spec to achieve the following: 1. Clearly identify and separate the two main elements of the spec: a. Framework (i.e. api/spi) b. Default implementation (i.e. .jam distribution format, tools, usage examples, etc.) 2. Ensure that the framework is the primary focus of the spec, identifying it as the means by which any implementation integrates with the SE compiler and runtime. 3. Clarify that any implementation will benefit from the integration story provided by the framework (I describe four concrete benefits in http://atsatt.blogspot.com/2008/04/jsr-277-could-be-great-for-osgi.html). 4. Clarify the rationale for the existence of a new implementation. 5. Recognize the importance of building a second implementation to validate the framework design. 6. Recognize OSGi as the right choice for that second implementation: a. A formal JCP adopted module standard. b. Wide market adoption. If the reasoning behind these proposals is not sufficiently clear to everyone, I'll be happy to elaborate. // Bryan
Re: Thread context class loader...
And my point remains, OSGi has always supported cycles and they have not been abused. -> richard Adrian Brock wrote: On Wed, 2007-12-05 at 16:48 +0100, Genevski, Pavel wrote: Hi Adrian, I am particularily interested in this statement: " * It isn't really that difficult to support " What do you exactly mean ? What is the current mechanism for supporting it ? I haven't looked at how Stanley supports them. Here's a flavour of the previous discussions (selected to answer your question) on the topic from my mail archive. Georgi and Richard discussing arguments against cycles and how hard they are to implement: " Danov, Georgi wrote: Hi, I know it is closed now, however, cycles are something too serious (for me) to let it just go : - I know very well the importance of backward compatibility and the pain of fixing your old mistakes that have become de-facto standards - the fact that SE itself has cycles should make it's developers think if they really need them. It would be interesting for me to understand, why do we have cycles in SE and what imposed their existence. - I am still behind the statement that cycles mean simply *bad* design, and I've witnessed how huge product was de-cycled successfully. - why are cycles bad? Because they create clouds of components that easily interconnect and create even bigger clouds. At the end, in order to stop single component, you need to bring the whole program/stack down. - in the scope of small to middle application, which does not need to be modular, cycles work and are not big problem. However, if you think about platforms, that provide runtime capabilities, cycles harm their operation. I am not trying to say that cycles are good, but I don't think these arguments against them are compelling. For example, any coupling among components, whether a cycle or not, creates a "cloud of components". I agree that cycles should generally be avoided, but supporting cycles is not the same as encouraging cycles. My proposal is that the runtime can run with cycle support, but by default this is off. This way legacy products will have alternative, plus affordable transition plan. Freshly developed components will not make use of cycles by default, so hopefully we see growing ecosystem of cycle-free components. Another, little more radical proposal is to allow cycles only for legacy jars that are wrapped as modules. Cycles between 'normal' modules should not be allowed. It seems like your fear is that by supporting cycles, then everyone will try to create cycles every chance they get. I don't think this is the case. OSGi has always supported cycles, but it is still exceptional to find the situation (even though it does happen). -> richard Georgi Danov NW JST Deployment Java EE 5 at SAP " Stanley and Richard discussing the complexity issue raised by an early draft reviewer: " I think we have briefly talked about cyclic dependency many months ago, but I don't think we have fully discussed it. The early draft currently assumes cyclic dependency is supported. NetBeans manages to work well without supporting cyclic dependencies, and same as .NET (Visual Studio detects and prevents circular dependencies to be created in assemblies). On the other hand, cyclic dependency is supported in OSGi, Ivy, and Maven. I agreed with the reviewer that supporting cyclic dependencies adds complexity to the module system, and many cyclic dependencies can be reduced by proper modularization. The fact that NetBeans and .NET manage to live without cyclic dependencies shows that it is possible to produce a module system with no cyclic dependency support that can still satisfy developers. On the other hand, while direct cyclic dependency can be detected and avoided, it may be more difficult for developers to avoid transitive cyclic dependency. I won't sit here and argue that cyclic dependencies are a must have feature, but I do think that they are not something that can always be avoided, especially in the transitive case and even in the case where you have closely related modules where you still want separation so that you can have alternate subsystem implementations and independent updatability. However, as someone who has implemented large portions of the OSGi R4 resolver algorithm, I can say that supporting cycles was not a significant source of complication in my code...that was actually the easy part. :-) -> richard " Glynn arguing for cycles such that modularization can be incremental: " A use case for cyclic dependencies is modularising a large, existing application. Cyclic dependency support makes it possible to adopt an incremental approach: 1. overlay existing components with modules and define or generate module dependencies based on the existing code dependencies
Re: Module system notification mechanism
I think your suggestion is reasonable. -> richard Stanley M. Ho wrote: Hi JSR 277 experts, Since we have been discussing some issues around the module instances' lifetime lately, I think it's probably a good time to bring up a related topic for discussion. As I reviewed the feedbacks from the EDR comments, from our previous discussions, as well as from my discussions with the teams in SE and EE, there were a few suggestions related to the module instances' lifetime: 1. The module system shall provide a way to monitor various events, e.g. module initialized, module released, etc. 2. The module system shall allow a module to have activator code. The activator code would be executed right before the module is fully initialized and when the instance is released. 3. A module shall have a way to be stopped. Having these suggestions don't mean we have to do all of them, and I would like to get your inputs. From my perspective, having a way to monitor module system's events (i.e. #1) seems very reasonable and useful, especially the use cases are very common. In fact, many teams in SE have expressed the needs in monitoring the module system's events in their class libraries in some degrees, so these libraries would react and behave appropriately. There are also other class libraries sitting on top of the JRE that have similar needs. For #2, this is a use case I gathered from EE, and this would be used mainly for registering and unregistering services when a module has been initialized or is released. Not that I think this is unimportant, but I am not yet convinced this is something we need to support directly at the module system level. For instance, if the module system notification mechanism (i.e. #1) is available, it should be possible for the EE server (or other apps that require similar functionality) to build a simple activator layer on top of the module system. For #3, the use case is that some EE servers might want to have the ability to "stop" a module by disabling the module's classloader when the module instance is released. In general, disabling a classloader is an uncommon and dangerous operation to perform, and it also violates the current classloading spec. While I agreed we should make this use case possible, I don't think this is something we want to push into the module system.; I believe there are alternatives we could consider to achieve the same result. For example, suppose there are APIs available to disable a classloader (might come from the classloading project) and the module system notification mechanism (i.e. #1) is available, it should be possible for the EE servers to hook into the notification mechanism, and disable the specific classloader it wants when a module instance is released from the module system. To keep things simple, I suggest we should support #1, and I hope this should be sufficient to enable other applications (e.g. EE servers) to support #2 and #3. I would like to hear what your thoughts are. - Stanley
Re: Relationship to JSR 291 [was: Re: Bryan's comments]
Stanley M. Ho wrote: Andy Piper wrote: ... 1. The ability to express a feature in terms of its constituent packages. Note that import-by-name doesn't really help here since the actual packages are distributed across multiple modules. What I really want is some kind of feature module that exports these packages to you when you ask for the feature. Unless I miss something, this is the main use case that import-by-name would address. Developers can package all the classes related to a feature (e.g. networking, xml, swing, ejb, etc.) into a module, and it exports (or reexports from other modules) all the things you need; other developers who want to use the feature would simply import that module by name. In fact, this is how we intend to use the module system to modularize various components in the SE platform. It certainly is possible to use the import-by-name approach to imitate some sort of import-by-feature approach, but this assumes that you are always able to create modules that have a one-to-one correspondence with some higher level "feature" concept. While this may work in some cases, e.g., replaceable modules implementing standard XML parsers, it is not likely to be possible in all cases -- the fact that it is the only mechanism available virtually guarantees that a majority of modules will not meet this criterion. Ultimately, this approach still falls short for the following reasons: * No matter how you describe it, the import-by-name approach == import-by-module, since name == module name in 277. Thus, you are still saying who gives you what you want, rather than what you want. * As a byproduct of the above, you are still defining dependencies too broadly, since import-by-name says that you need everything from this named module, even though you might only need some very small subset, which will lead to dependency fanout. Further, I would say that it is more wishful thinking rather than actual reality to believe that a higher level concept even helps here, since in the end for Java it always comes down to importing packages and using them in your source code. The higher level concept does not exist in source code nor does the programmer know anything about it; the programmer does, however, know about packages and the associated API since this is required to use anything in Java. The real value of this sort of higher level concept is further diminished when you introduce tooling support that can pull these sorts of dependencies from the byte code and automatically generate dependency metadata, such as BND does for OSGi bundles. People may like the "import-by-name" approach when they are developing and need to create a class path for their IDE, but in reality there is no reason why they should have to do this for their IDE either. Assuming that a module repository exists, if I start using classes from Hibernate, for example, there is no reason why my IDE cannot see that this package is not in my class path and offer to fix it by searching the module repo for providers of the Hibernate package, similar to how IDEs currently fix imports in the source code...in this case it could even allow you to select from multiple versions. Then the issue of "ease of use" goes away completely. So, in short, call it what you want, but the solution offered by 277 is import-by-module with all of the associated shortcomings. -> richard
Re: Relationship to JSR 291 [was: Re: Bryan's comments]
Andy Piper wrote: At 00:53 08/06/2007, Bryan Atsatt wrote: I truly believe that the Java world will be far better served by having 277 support ONLY import-by-package! But there has been this background fear that import-by-package is somehow terribly complex. I can certainly believe that the union of all the OSGi features *does* result in a lot of complexity. Just to express a usage datapoint here. Having used OSGi for a while now I am finding import-by-package too hard to use in certain circumstances (but in general it is a nice flexible model), but its not an issue that import-by-name really fixes in my opinion. The basic issue is that to use something I am interested in the union of packages that make up the "feature". I actually don't care about the individual packages. Take my current bugbear, cglib in Spring for example, to use this "feature" the *proxied* class bundle needs to import the following packages: net.sf.cglib.proxy, net.sf.cglib.core, net.sf.cglib.reflect, org.aopalliance.aop, org.springframework.aop.framework, org.springframework.aop, not an obvious set, nor a set of packages that I care much about. Worse, the proxied class does not know ahead of time that it needs to import these packages (this is I believe because cglib tries to create the proxy in the classloader of the proxied class by default). Ideally to solve this I would like two things: 1. The ability to express a feature in terms of its constituent packages. Note that import-by-name doesn't really help here since the actual packages are distributed across multiple modules. What I really want is some kind of feature module that exports these packages to you when you ask for the feature. 2. Some way of augmenting a module *after* the fact with a particular set of packages, so that the module doesn't need to do dynamic-import * everywhere. I don't think either if these is possible in OSGi right now (I would be happy to be corrected!). I don't think JSR 277 solves these either (but again I would be happy to be corrected). Yes, you are correct that both of these are not currently possible...well, I guess (2) is technically possible since you could grab the original bundle JAR file and rewrite its manifest then install it, but I know this is not what you want. Both of these have been discussed at one point in time or the other. Peter Kriens was recently talking to me about something similar to (1), so we might be doing some experiments in this area. I think something like (1) could fit in the OSGi model fairly well, but more investigation is needed to be sure. -> richard Just a datapoint from real-world usage. andy Notice: This email message, together with any attachments, may contain information of BEA Systems, Inc., its subsidiaries and affiliated entities, that may be confidential, proprietary, copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named in this message. If you are not the intended recipient, and have received this message in error, please immediately return this by email and then delete it.
Re: Relationship to JSR 291 [was: Re: Bryan's comments]
Stanley M. Ho wrote: Hi Richard, Richard S.Hall wrote: ... Perhaps so. I am operating off my [possibly dated] recollection that 277 modules explicitly list the classes that they export, which meant that they may expose arbitrary public classes from a package. If this is no longer the case, then this is fine. The point remains, all legacy OSGi dependencies on a package assume that all public classes are exported from that package. An OSGi dependency on a given package cannot be resolved to a 277 module whose "exported public" classes are not the same as all "public" classes in the specific package. I was under the impression that the class filtering is also in effect to determine which public classes are exported from that package in OSGi and that the OSGi dependency on a package already has a similar "exported public" concept. This is definitely true, but this was only introduced in R4 and is still intended to be the exception, not the norm...from my understanding, 277 defined the norm in the opposite direction. Ultimately, if we are making the same assumption, which is that people will generally be exporting all public classes from a package and only occasionally filtering some classes that should not be public, and we are promoting this as the best practice, then we are at least in the same ball park. Anyway, these are all related to how the OSGi framework might import 277 modules by package name. I think it would be good to know if this is actually a sensible thing that the OSGi framework wants to support before we dive too deep on this. Glyn/Richard, is this something that you foresee the OSGi framework will support? Well, effectively all OSGi dependency resolution comes down to packages at some point, so if we are going to use 277 modules at all in the OSGi framework, then we have to have some way to determine their set of exported packages, which is necessary for determining consistency. -> richard
Re: Relationship to JSR 291 [was: Re: Bryan's comments]
Bryan Atsatt wrote:
Stanley M. Ho wrote:
Glyn Normington wrote:
*Bryan Atsatt <[EMAIL PROTECTED]>* wrote on 30/05/2007 07:57:59:
...
> So the open issue is the richness of the import "language": must we
> support only lowest-common-denominator, or can we do better without
> over-complicating the design?
>
> I for one would like to be able to have a single module express
> dependencies on modules from both the same and different
ModuleSystems,
> *using the standard semantics of each*. This may be reaching too
far,
> but we should at least explore it seriously while we figure out what
> interop means here...
At this point, I feel that is likely to be reaching too far, but I'm
happy to play along and see what we can learn along the way.
I agreed with Glyn that this might be reaching too far. Before we dive
too much into how to implement interoperability, I think one of the
outstanding questions we should answer first is what degree of import
interoperability we want to offer. There are four possibilities:
1. JSR 277 module imports OSGi module by module name
2. OSGi module imports JSR 277 module by module name
3. OSGi module imports JSR 277 module by package name
4. JSR 277 module imports OSGi module by package name
Let's ignore the 294 issues and the module initialization issues for now
to simplify this discussion.
I think we all agreed #1 and #2 are important to support and the
reflective APIs already enable these (of course, there are minor issues
we still have to address as we evolve the APIs.)
That said, it is unclear to me how important it is to support #3 and #4,
so I think the first question for this EG is whether we want to support
#3 and #4 at all. However, let's pretend #3 and #4 are important for
this specific discussion.
I think #3 is already possible since the OSGi framework could look up
the appropriate module which exported the package in the repository,
using the reflective APIs. Afterwards, the OSGi framework can then do
the necessary wiring. Yes, the reflective APIs can be further refined to
make it easier for the OSGi framework to query modules by exported
package name from the repository, but this is a minor issue that we can
address easily.
#4 is a bit complicated, because there is no import-by-package semantic
in the current module layer's APIs and I think we all agreed that we
don't want to support this semantic directly in the module system
defined by JSR 277.
Not me.
I do understand your point that this is more difficult, but I don't
really care--it is our job to do the Right Thing here. Sure, if it is
not feasible in the time frame we have, then so be it. But we haven't
even tried!
If we're forced to make a choice between the two models, I'd pick
import-by-package with zero hesitation.
OSGi has a lot of experience here, and we shouldn't ignore it.
Import-by-name was added late in the game, and is now seriously
downplayed, for good reason.
Just for information, import-by-name (or module dependencies) was not
added to OSGi R4 because they were thought to be widely needed or
useful, adding them was always somewhat contentious which is why the R4
spec contained a section saying why you shouldn't use them (this section
has been further expanded in R4.1). The main motivation for adding them
was certainly legacy situation and for tightly coupled sub-systems.
All in all, I agree with everything that Bryan is saying and I argued
these same points early on; however, I was under the impression that we
had long since lost that battle.
Bryan, it is unclear to me whether your view is that we need to make
sure that 277 supports other modules systems the use import-by-package
or if you think that 277 should somehow directly support
import-by-package (i.e., explicitly expose such concepts).
-> richard
And Oracle has a lot of relevant experience as well, since the shared
loader mechanism I created for our AS stack ("shared libraries")
supports only the import-by-name model. We have *lots* of
shared-libraries (I've seen 100+ running simultaneously), used by many
different components in the stack, in addition to customer applications.
It is a tremendous pain when refactoring or even simple renaming has to
be done (I hope to shift this implementation onto 277 to solve this
problem, rather than inventing an interim approach). And this kind of
change happens far more frequently than you probably expect.
I don't want to repeat the same mistake here. Nor do I think that
leaving a long trail of extraneous "view" modules behind for
compatibility is a good solution.
Worse still, if the 277 APIs don't support import-by-package, then the
compiler won't support it either. If the compiler doesn't support it,
then we've lost a golden opportunity to move to a better world, one in
which packaging is irrelevant.
We're supposed to be eliminating "jar-hell" here. Let's not simply
replace it with "module-hell".
In this context, the question is really about what
it would take to mak
Re: Relationship to JSR 291 [was: Re: Bryan's comments]
On May 30, 2007, at 5:55 PM, Stanley M. Ho wrote: Glyn Normington wrote: *Bryan Atsatt <[EMAIL PROTECTED]>* wrote on 30/05/2007 07:57:59: ... > So the open issue is the richness of the import "language": must we > support only lowest-common-denominator, or can we do better without > over-complicating the design? > > I for one would like to be able to have a single module express > dependencies on modules from both the same and different ModuleSystems, > *using the standard semantics of each*. This may be reaching too far, > but we should at least explore it seriously while we figure out what > interop means here... At this point, I feel that is likely to be reaching too far, but I'm happy to play along and see what we can learn along the way. I agreed with Glyn that this might be reaching too far. Before we dive too much into how to implement interoperability, I think one of the outstanding questions we should answer first is what degree of import interoperability we want to offer. There are four possibilities: 1. JSR 277 module imports OSGi module by module name 2. OSGi module imports JSR 277 module by module name 3. OSGi module imports JSR 277 module by package name 4. JSR 277 module imports OSGi module by package name Let's ignore the 294 issues and the module initialization issues for now to simplify this discussion. I think we all agreed #1 and #2 are important to support and the reflective APIs already enable these (of course, there are minor issues we still have to address as we evolve the APIs.) That said, it is unclear to me how important it is to support #3 and #4, so I think the first question for this EG is whether we want to support #3 and #4 at all. However, let's pretend #3 and #4 are important for this specific discussion. I think #3 is already possible since the OSGi framework could look up the appropriate module which exported the package in the repository, using the reflective APIs. Afterwards, the OSGi framework can then do the necessary wiring. Yes, the reflective APIs can be further refined to make it easier for the OSGi framework to query modules by exported package name from the repository, but this is a minor issue that we can address easily. I am not so sure supporting #3 is as straight forward as you suggest here. Specially, the OSGi modularity layer [for the most part] assumes that packages are atomic units; there is no way to specify that your export or import something smaller than a package. Since 277 modules declare exports in terms of classes, it is not clear how we would know if a 277 module was exporting an entire package. This would, at a minimum, require that we assume we can inspect the contents of the module and see that the contents of a contained package matches the module's export signature for that package. -> richard #4 is a bit complicated, because there is no import-by-package semantic in the current module layer's APIs and I think we all agreed that we don't want to support this semantic directly in the module system defined by JSR 277. In this context, the question is really about what it would take to make #4 possible if we want to support it, and so far there are two different approaches we have discussed: a. Treat import-by-package separately from import-by-module in the reflective APIs and make the module layer fully aware of the import-by-package concept. b. Expose exported-package using the ModuleDefinition abstraction, and provide necessary hooks (e.g. consistency checking) for the module system to resolve this kind of dependency appropriately. The module layer is not aware of the import-by-package concept at all. New import dependency granularity could also be introduced in a similar manner in the future without requiring significant changes in the module layer. As Richard and I discussed in previous emails, it seems to make the most sense for JSR 277 to define the minimal set of features possible in the module layer to accomplish what needs to be accomplished to provide core modularity support in the Java platform. I think (b) follows this principle nicely while (a) does not. If we decide to support #3 and #4, then how we want to address #4 would impact our overall design significantly. Is there anyone in the EG provide good reasons why #3 and #4 are important to be supported? If so, is there anyone in the EG think (b) is not feasible at all and we should do (a) instead to support #4? Can you explain your rationale? > RESOLUTION MODELS > > The current design requires that each ModuleSystem provide its own > resolution logic, and that each definition will be resolved by its > owning ModuleSystem. This model appears to provide flexibility for > significant differences in implementation, but we really don't know > enough at this point. Perhaps only an actual second implementation will > tell us if this provides useful flexibility. In the existing module systems we are aware, e.g. OSGi and NetBeans, we alread
Re: Service providers
Hey Stanley, Stanley M. Ho wrote: Richard S. Hall wrote: Hello Stanley, ... I guess the addition of service-related concepts into the module layer would be one example of feature creep for me. While I think it makes perfect sense to figure out how the Service Loader stuff will work on top of the module layer, pushing Service Loader concepts down into the module layer doesn't make any sense to me at all. It should be sufficient for the Service Loader to probe the installed modules and perhaps examine module metadata to determine if a module provides a service or not. From there, the Service Loader can create module instances and provider instances as necessary. If this is not possible in our current constructs, then we should address these shortcomings rather than adding higher layer concepts into the module layer. -> richard The current service provider strawman suggests a few changes in ModuleDefinition/Query classes, and you basically suggest that these changes could be eliminated if we have a more generic way to express services and service-providers in the module metadata and a generic way to examine the information from the module metadata, all without service-related APIs in the module layer. I think what you suggested makes sense, and I will look into this. To me, this sounds like what we call the "extender model" for the OSGi framework. The way it works is that bundles that want to participate in certain scenarios simply include some metadata inside of themselves, then some infrastructure can probe for this metadata and automatically do work on their behalf. This is how Declarative Services works in the R4 spec and also how Spring-OSGi works...it is becoming a very common model. The benefit of this approach is that it doesn't push upper layer concepts down into the modularity layer and keeps it simple. Off the top of my head, I don't see a reason why the Service Loader couldn't use a similar approach. -> richard
Re: Relationship to JSR 291 [was: Re: Bryan's comments]
Hello Stanley, Stanley M. Ho wrote: Richard S. Hall wrote: Agreed, but I think part of the reason that there is an urge to push such features into the core is because there is already feature creep in the core that gives people the impression that we are redefining OSGi modularity, NetBeans modularity, etc. Thus, everyone wants to make sure their pet feature is added. If 277 could make a stand and declare that it is at a layer beneath all of that (and live up to that declaration), then there would likely be less push down. Richard, It is unclear to me what you meant by feature creep in the core because everything in the specification is essentially required to address the problems this JSR is set out to solve. We can argue that some features should not be in the base layer API but in the module system defined by JSR 277 instead or vice versa, but they still belong to the overall JSR 277 specification. I guess the addition of service-related concepts into the module layer would be one example of feature creep for me. While I think it makes perfect sense to figure out how the Service Loader stuff will work on top of the module layer, pushing Service Loader concepts down into the module layer doesn't make any sense to me at all. It should be sufficient for the Service Loader to probe the installed modules and perhaps examine module metadata to determine if a module provides a service or not. From there, the Service Loader can create module instances and provider instances as necessary. If this is not possible in our current constructs, then we should address these shortcomings rather than adding higher layer concepts into the module layer. -> richard To recap what we have been discussing, I think our general consensus is to define a module system in JSR 277 that would address all the problems this JSR is set out to solve, as well as to provide a high level framework for different module systems (including the module system defined by JSR 277) to layer on top for interoperation. We only described the former in the EDR, but the latter should become more evident in the next revision of the JSR 277 specification. - Stanley
Re: Relationship to JSR 291 [was: Re: Bryan's comments]
Stanley M. Ho wrote: Hi Bryan, Bryan Atsatt wrote: Let me clarify. It is my assumption that 277 will provide both an API/framework *and* a specific ModuleSystem implementation. And that OSGi will be able to supply another ModuleSystem implementation: 277 API/Framework / \ / \ 277ModuleSystem OSGiModuleSystem Right. There are also other module systems that can be plugged into this framework, e.g. NetBeans. Therefore, the API's must *support* "import package", or it will not be usable by the OSGiModuleSystem. I agree that the 277ModuleSystem implementation does not need to support "import package" semantics. Which means that the 277 APIs must support *both* semantics. Otherwise, OSGi will not be able to operate as a first-class citizen in the 277 world. And, if 294 adds support for "import", it should at least do the natural thing of "import package". And it should also support "import module", but supporting both at the language level may be awkward. // Bryan I think we all agreed that the module system defined by JSR 277 does not need to support import package. As far as I am aware (correct me if I'm wrong), all the existing systems support import at the module level, and OSGi is the only one that supports import at the package level. Thus, import by package should be a feature specific to the OSGi specific module system implementation (e.g. OSGiModuleDefinition), and I don't think it is necessary to push it into the base classes in JSR 277. Well, I must at least say that this is flawed logic at best. The fact that only the OSGi framework supports importing packages does not say anything about whether or not it should be supported by 277 or not. There any many valid arguments why the import package model is better due to the fact the is declares "what" is needed, not "who" contains what is needed. Using "who" instead of "what" generally leads to terrible dependency fanout due to coarse-grained module definitions and brittleness. I am fairly certain that I could conjure up some people who could tell you stories about the horrors of the brittleness of the module dependency approach when trying to re-use modules for different purposes. However, I guess this is beside the point... I think the real argument that should be made for not necessarily including import package support is along the lines of what I was saying in my other message, which is that we should keep 277 as simple as possible. This doesn't mean supporting the notion of import by package is impossible. What OSGi will expose from its repository is simply module definitions, and module definition is basically an abstraction. Therefore, it should be possible to treat an OSGi bundle as a module definition, as well as to treat each exported package in an OSGi bundle a module definition as well. The dependency supported in JSR 277 is import-by-module. In the JSR 277 module system, this is mapped to import-by-superpackage. In the OSGi module system, this can be mapped to import-by-OSGi-bundle or import-by-package. If other module systems want to support different import granularity or if OSGi evolves to add new import granularity in the future, this model should still work. Honestly, interoperability between multiple module systems with "import module" alone is already a hard enough problem for us to solve. I am concerned that we're making this problem towards insolvable if we keep adding complexity. ;) Agreed, but I think part of the reason that there is an urge to push such features into the core is because there is already feature creep in the core that gives people the impression that we are redefining OSGi modularity, NetBeans modularity, etc. Thus, everyone wants to make sure their pet feature is added. If 277 could make a stand and declare that it is at a layer beneath all of that (and live up to that declaration), then there would likely be less push down. -> richard
Re: Another reminder: Updated JSR-277 specification (04/19/2007)
Bryan Atsatt wrote: Hey Stanley, More comments inline. // Bryan Stanley M. Ho wrote: Hi Bryan, Thanks for the inputs. See my comments inline. Bryan Atsatt wrote: Stanley, Here are my comments on both the spec and the apis. // Bryan SPEC 1.5 You are assuming that module-name==superpackage-name, right? Otherwise, if we add import statements to 294, then these would have to specify module names, thus creating a circular dependency. Right. I will clarify it in the next revision of the spec. 2.1 "is properly understood as a mathematical abstraction" This phrase seems unnecessary, and comes across as a bit highbrow to me. "A module archive is inherently sealed, and it does not reference any resource externally" The term "sealed" here could be confused with the manifest attribute. Perhaps a better term is "self-contained". And the second phrase is a bit confusing also, since imports do reference external "resources". Perhaps if you said "may not directly reference external classes or resources"? I will look for better words to describe it. 2.2 Should state that module name must == superpackage name. 2.4 The members definition should be a list of *packages*, not classes. That is all the 294 proposal supports (I actually can't see it at the moment since my HD died, but this is my recollection!). I think it is reasonable to assume it will stay this way. The member declared in the source file is indeed a list of packages. That said, based on my understanding of 294, the compiler would expand the member packages to a list of classes in the artifact. I will clarify it in the spec. Version may be optional in a superpackage, but it is mandatory for a module, right? This should be clarified. "mandatory" would mean something that we can check and enforce. Since version is declared using annotation in a superpackage, I don't think we can enforce it at build time. That said, we can still enforce it at deployment time if this is what you meant. Yes, the 277 spec should describe it as mandatory, and we should enforce it at runtime. If it becomes possible to enforce it a compile time, we would do so. The example members list should be package names only. 2.7.3 I like this addition. I'm glad to know. 2.7.4 There may well be cases in which a module wants to re-export a subset of imported classes/resources. We should consider supporting this case. Could you describe the actual use cases for this? Package a.b contains and exports classes C and D. Package x.y imports a.b, but only wants to re-export C. I am not sure that this case is such a good idea. By allowing it, wouldn't consumers of package x.y then potentially be able to see a D from some other provider...this seems like a recipe for subtle bugs. You mention the OSGi model below, but such subsetting of packages really breaks the OSGi model of atomic packages. Refactoring in this way results in a perhaps unexpected runtime/memory overhead. A pure wrapper module that re-exports must have its own class loader, and, even though it won't define any classes, the VM *cache* will be updated as that loader is used. I think we need to think hard about this issue. The OSGi model of import by *package name* decouples the importer from any explicit binding to a bundle/module name. Refactoring under that model is *much* cleaner, and far more natural. As is the usage model. After all, Foo.java import statements contain package/class names, *not* module names. Programmers think in terms of classes and packages. Peter makes this point pretty strongly, and I have to say I agree wholeheartedly: http://www.aqute.biz/Blog/2006-04-29 Me too. -> richard 4.1 Since not everyone uses the Sun terms (FCS, GA), we might want to also define qualifier here as "beta", or "pre-release", or "release-candidate". And a "release" version must not have a qualifier. I will clarify it. 5.2.2 We should also make it clear that resources will also often be contained in "legacy" jars, for which no ClassesDirectoryPath is required. There is related to an open issue on how we want to support "legacy" jars in the implementation. I will open up a new thread for the discussion on this topic. 5.6 Might be a good idea to explicitly state that the Class-Path attribute is ignored. Right. #3 This would appear to rule out OSGi's support of split packages. This spec should enable each ModuleSystem to control this behavior. The JAM file is the distribution format for the module system defined by 277, and I don't think the treatment of JAR manifest in this section applies to OSGi. Sure, for the manifest of a .jam file. But the statement: "All packages defined in a module definition are inherently sealed, and this entry is ignored by the module system." is pretty broad, and would seem to indicate that it applies to the definitions produced by any module system. 6.1 Third paragraph: Remove the leading "Besides, " 6.2.2 Repository shutdo
