Re: Loading an automatic module from an exploded directory
> > Directories with a ".jar" suffix is clever but I don't think it gets away > from the substantive issue that Java SE only specifies how automatic > modules come into being when they are packaged as JAR files. I could also > imagine tools and libraries that scan tripping up on directories named > ".jar". There are also many libraries that peek into the JAR manifest with > the java.util.jar APIs so they will be surprised too. > I'm sure we could agree on some other convention to stop consumers from tripping on ".jar" in a directory name. (They should check ZipEntry.isDirectory, anyways) Just on MultiModuleJARs (and its buddy #MultiModuleExecutableJARs). This > would probably be significant enough to need a JEP. It would need to > explore whether the modules in the JAR are observable when the JAR file is > on the module path, this amounts to deciding if they are resolved and > loaded into the boot layer or child later. If observable on the module > path then it mean javac would need to support it. Automatic modules would > bring many discussion points to the table. One of their important roles of > automatic modules is to bridge to the class path so what does not mean when > everything is packaged into the same JAR file, does the JAR file have a > directory with the "class path" too? There are implications for jlink and > several other tools (jar) aswell. So in summary, and at least in my view, > there is a lot more exploration needed before doing just implementation > changes to support "exploded automatic modules" in JAR files. > Ok, so #MultiModuleJARs obviously has a much broader scope than what I originally thought. I'm not suggesting that the semantics of this JAR file I produce should change. The JDK should remain blissfully unaware that I packed modules somewhere deep into /META-INF/. I don't expect javac, jlink, the java command line (or java -jar) to find these modules, nor know about their existence. Loading these bundled modules would require rolling your own custom main method using ModuleLayer and friends to set this up. I'm not suggesting we standardize how this loading should happen. All (I think) I'm suggesting is a local fix in ModulePath to allow reading automatic modules from ZIP filesystem paths given that some specified convention signals that this path represents an unpacked dusty jar. This convention would need to be documented (in ModulePath?), but that should represent a small javadoc change. The semantics of what an automatic module is would not change. ModulePath already supports reading proper modules from such ZIP paths, so one can claim that not supporting automatic modules represents a bit of an asymmetry. I think this change unlocks potentially useful new distribution models for modules, and if #MultiModuleJARs was to be revisited, something like this would probably be required as a building block anyway. Cheers, Eirik.
Re: Loading an automatic module from an exploded directory
On 18/04/2020 23:04, Eirik Bjørsnøs wrote: : Could we update ModulePath could to detect "automatic module directories"?. Any directory path name ending with ".jar" could be loaded as an automatic directory module, with the same semantics as if the directory was zipped into a JAR. This would require a refactoring of deriveModuleDescriptor to allow it being called both from readJar and readExplodedModule. Then readExplodedModule would need an update to detect the lack of module-info.class, and call deriveModuleDescriptor just like readJar does. scan(Path) would need an update to make it skip directory scanning on directories ending with ".jar". As far as I can tell, this would be (or allow) an implementation of MultiModuleJARs. Directories with a ".jar" suffix is clever but I don't think it gets away from the substantive issue that Java SE only specifies how automatic modules come into being when they are packaged as JAR files. I could also imagine tools and libraries that scan tripping up on directories named ".jar". There are also many libraries that peek into the JAR manifest with the java.util.jar APIs so they will be surprised too. Just on MultiModuleJARs (and its buddy #MultiModuleExecutableJARs). This would probably be significant enough to need a JEP. It would need to explore whether the modules in the JAR are observable when the JAR file is on the module path, this amounts to deciding if they are resolved and loaded into the boot layer or child later. If observable on the module path then it mean javac would need to support it. Automatic modules would bring many discussion points to the table. One of their important roles of automatic modules is to bridge to the class path so what does not mean when everything is packaged into the same JAR file, does the JAR file have a directory with the "class path" too? There are implications for jlink and several other tools (jar) aswell. So in summary, and at least in my view, there is a lot more exploration needed before doing just implementation changes to support "exploded automatic modules" in JAR files. -Alan
Re: Loading an automatic module from an exploded directory
On 10/04/2020 17:48, Eirik Bjørsnøs wrote: I once had the pleasure of debugging a system which after running in production for a few days would suddenly start returning 404 errors. Turned out someone (me) had forgotten to specify where Jetty should unpack its WAR files. So they ended up on shared temp space where a weekly Centos cron job cleaned them up. After this experience I pledged to never unpack code to temp again :-) Also, the performance implications seems not very appealing. If MultiModuleJARs [1] is re-visited then it would need to explore this topic a bit more. For now, the only case where the JDK implementation unpacks is when ModuleFinder.of is created with a file path to JAR file in a custom file system or to a directory in a custom file system that contains JAR files. These JAR files might be signed so this is why JarFile (with the JAR verifier support) is used. It would be possible to do the verification without support for the JarFile infrastructure, it just wasn't a priority in JDK 9. -Alan [1] https://openjdk.java.net/projects/jigsaw/spec/issues/#MultiModuleJARs
Re: Loading an automatic module from an exploded directory
On Fri, Apr 10, 2020 at 5:00 PM Alan Bateman wrote: > > This would require the API specify how automatic modules are created > from the contents of "something" that already have a ModuleReader. > Specification could be crafted of course but I have concerns that it > goes beyond the original intention of automatic modules. That is, the > original intention was to allow yesterday's JAR files be used as today > modules. More sophisticated applications have the APIs to create > automatic modules for other cases where needed. So I think it needs a > bit more thought on whether additional APIs should be exposed. > Fair enough. The use case and discussion is noted here for posterity if similar needs should appear in the future. Thanks for your thought and input, Alan. Eirik.
Re: Loading an automatic module from an exploded directory
> > Do they need to be "exploded"? If the dependences are automatic modules > then I assume putting the JAR files (without unpacking) into > META-INF/modules should just work without needing a new ModuleFinder > implementation or implementing the exploded equivalent of multi-module JAR > files (or modular multi-release JAR files). > I once had the pleasure of debugging a system which after running in production for a few days would suddenly start returning 404 errors. Turned out someone (me) had forgotten to specify where Jetty should unpack its WAR files. So they ended up on shared temp space where a weekly Centos cron job cleaned them up. After this experience I pledged to never unpack code to temp again :-) Also, the performance implications seems not very appealing. > It does this in order to support signed JARs as the zip file system > provider doesn't have any support for validating signatures. > Hmm.. I guess exploded jars won't support signature verification. Too bad that the implementation is so tightly coupled to jar files. It could instead have defined behaviour in terms of abstract paths within a package, in which signatures would 'just work'. Similar situation for automatic modules. That could also have been specified in terms of abstract paths without mandating a specific packaging format. > Understood although you probably don't need a deep copy of everything as > there is a lot in exploded module reader related to potential security and > traversal on the default file system that is less interesting in JAR files. > Thanks, this code did seem security sensitive, so I conservatively did a deep copy. I have pruned some of the code now. Redeploying the module containing the JAX-RS resource takes something like > 50ms IIRC. > > This sounds very interesting. I'm sure there are several people here that > would be interesting in seeing a write-up or demo of this. I > I'll see if I can get a demo together at some point. > I'm curious if the service wiring make use of the existing uses/provides > or something else. > Modules make use of provides to expose binding classes to the runtime. Binding classes contain methods which consume and produce services, thereby forming a dependency graph. > I'm also curious if you've run into any issues with multi-parent > configurations as that is an area that might need attention some day. > The Jersey layer in the demo depends on both the servlet layer and the jax.rs layers. However, these two layers do not share any modules, so I didn't get into trouble. One could imagine a layer depending on two parent layers who define separate "instances" of the same module, which I think JPMS would flag as an invalid configuration. My Maven plugin already excludes modules from a layer when the dependency is already visible up the ancestor chain, so it should be straightforward to add a check for detecting same-module-in-multiple-parents and flag that to the user at build time. Eirik.
Re: Loading an automatic module from an exploded directory
On 10/04/2020 11:14, Eirik Bjørsnøs wrote: : Since ModuleReader already has the list() method, code in java.lang.module could use that to scan for packages and service names it needs to derive automatic ModuleDescriptors. Clients would now only need to pass the custom module reader and a default automatic name: ModuleFinder moduleFinder = ModuleFinder.of(ModuleReader moduleReader, String defaultAutomaticModuleName); I would still have to create my custom module reader, but I would no longer need to know or care about the details of how automatic modules work. Something to consider? This would require the API specify how automatic modules are created from the contents of "something" that already have a ModuleReader. Specification could be crafted of course but I have concerns that it goes beyond the original intention of automatic modules. That is, the original intention was to allow yesterday's JAR files be used as today modules. More sophisticated applications have the APIs to create automatic modules for other cases where needed. So I think it needs a bit more thought on whether additional APIs should be exposed. -Alan.
Re: Loading an automatic module from an exploded directory
On 10/04/2020 10:53, Eirik Bjørsnøs wrote: : Is it safe to assume that all access to content with a module happens through the ModuleReader? Yes, the ModuleReader is the only way to access the contents of a module so you'll see all class loading, finding resources, etc. all go through the reader. We can't stop nosey code opening the packaged modules directly of course but none of the standard APIs bypass the module reader. -Alan
Re: Loading an automatic module from an exploded directory
On 09/04/2020 19:23, Eirik Bjørsnøs wrote:
Alan,
If I read your mail correctly, you are creating "multi-module JAR
files"
where the modules are "exploded" under /META-INF/modules in
${NAME}-${VERSION} directories.
Correct.
Do they need to be "exploded"? If the dependences are automatic modules
then I assume putting the JAR files (without unpacking) into
META-INF/modules should just work without needing a new ModuleFinder
implementation or implementing the exploded equivalent of multi-module
JAR files (or modular multi-release JAR files).
(The reason it should work is that the built-in implementation can work
with paths to locations in virtual file systems. It handles this by
copying the JAR files from the virtual file system to the default file
system so that it can be opened with the JarFile API. It does this in
order to support signed JARs as the zip file system provider doesn't
have any support for validating signatures. It's possible this approach
isn't very efficient, also we haven't looked into interactions with the
cron jobs that clean tmpfs so there may be a bit more work needed here).
To achieve this, I had to copy / replicate quite a bit of code from
the ModulePath class which is not accessible outside java.base.
It wasn't all that hard, I just wasn't happy with the amount of
classes / methods I had to copy from java.base only to slightly modify
them
This includes:
Code to create / build the ModuleDescriptor (stolen from
deriveModuleDescriptor)
Code to clean the artifactId used to produce the automatic name
(Stolen from cleanModuleName, Patterns and Checks)
Duplication of ExplodedModuleReader which also dragged in Resources.
Understood although you probably don't need a deep copy of everything as
there is a lot in exploded module reader related to potential security
and traversal on the default file system that is less interesting in JAR
files.
:
The runtime loads a graph of module layers, each containing one or
more modules. Layers may depend on other layers, forming a DAG with
corresponding class loader delegation.
When the runtime detects that a module needs to be redeployed, the
transitive closure of depending layers is removed from the graph and
new module layers are created and added to the graph.
Services are bound using dependency injection, so there's also a DAG
of service dependencies. This allows restarting services which are
affected by the redeploy, either because they are provided from
updated modules or because they consume services from updated modules.
So not really hot-deploying in the JVM sense, but still pretty fast
and developer-friendly.
My demo has a module which provides a JAX-RS resource which is
consumed by a Jersey module which provides an HttpServlet which is
again consumed by a Jetty module which deploys the servlet in a
servlet context.
Redeploying the module containing the JAX-RS resource takes something
like 50ms IIRC.
This sounds very interesting. I'm sure there are several people here
that would be interesting in seeing a write-up or demo of this. I'm
curious if the service wiring make use of the existing uses/provides or
something else. I'm also curious if you've run into any issues with
multi-parent configurations as that is an area that might need attention
some day.
-Alan
Re: Loading an automatic module from an exploded directory
On Thu, Apr 9, 2020 at 5:42 PM Eirik Bjørsnøs wrote: Here's a strawman API which I think would support my use case : > > ModuleFinder auto = ModuleFinder.automatic(Path location, String > defaultName, Set packages, Set serviceNames) > Another thought... Since ModuleReader already has the list() method, code in java.lang.module could use that to scan for packages and service names it needs to derive automatic ModuleDescriptors. Clients would now only need to pass the custom module reader and a default automatic name: ModuleFinder moduleFinder = ModuleFinder.of(ModuleReader moduleReader, String defaultAutomaticModuleName); I would still have to create my custom module reader, but I would no longer need to know or care about the details of how automatic modules work. Something to consider? Cheers, Eirik.
Re: Loading an automatic module from an exploded directory
On Thu, Apr 9, 2020 at 7:37 PM Alan Bateman wrote: The only packaging format for automatic modules that Java SE defines is > JAR files. The "Multi-release JAR files" feature is also JAR file only. > My ModuleFinder for exploded-within-jar automatic modules needed a ModuleReader implementation. I decided to make that ModuleReader multi-release-aware. Like JarFile, it inspects the manifest for the Multi-Release attribute and scans META-INF/version directories accordingly. Is it safe to assume that all access to content with a module happens through the ModuleReader? Eirik.
Re: Loading an automatic module from an exploded directory
Alan,
> If I read your mail correctly, you are creating "multi-module JAR files"
> where the modules are "exploded" under /META-INF/modules in
> ${NAME}-${VERSION} directories.
Correct.
> It shouldn't be too hard to create your
> own ModuleFinder that finds modules under META-INF/modules. This would
> mean implementing ModuleFinder rather trying to use
> ModuleFinder.of(Path...).
This is what my implementation currently does.
To achieve this, I had to copy / replicate quite a bit of code from the
ModulePath class which is not accessible outside java.base.
It wasn't all that hard, I just wasn't happy with the amount of classes /
methods I had to copy from java.base only to slightly modify them
This includes:
Code to create / build the ModuleDescriptor (stolen from
deriveModuleDescriptor)
Code to clean the artifactId used to produce the automatic name (Stolen
from cleanModuleName, Patterns and Checks)
Duplication of ExplodedModuleReader which also dragged in Resources.
I assume you've found ModuleDescriptor.read to
> read/parse the module-info.class of explicit modules.
I don't actually need to read the explicit modules. I just check that they
have module-info.class and use let ModuleFinder.of do the rest of the job.
> You are right that
> it would require code to scan directory trees, at least the equivalent
> of automatic modules, maybe for explicit modules too. However, it
> shouldn't be too hard. Have you tried the zip file system provider? That
> would allow you to open the JAR file as a file system so you can use the
> file system API.
>
>
Yes, I'm opening a ZIP filesystem on the single-jar and using the file
system API to navigate it.
> > I have also identify an additional use case which is to allow
> hot-deploying
> > automatic modules during development from target/classes using a Maven
> > plugin.
> >
> I'm not sure how to interpret this but just to say that the unit of
> replacement is the module layer, you can't replace modules in a layer
> and/or dynamically change the set of packages in a loaded module.
>
The runtime loads a graph of module layers, each containing one or more
modules. Layers may depend on other layers, forming a DAG with
corresponding class loader delegation.
When the runtime detects that a module needs to be redeployed, the
transitive closure of depending layers is removed from the graph and new
module layers are created and added to the graph.
Services are bound using dependency injection, so there's also a DAG of
service dependencies. This allows restarting services which are affected by
the redeploy, either because they are provided from updated modules or
because they consume services from updated modules.
So not really hot-deploying in the JVM sense, but still pretty fast and
developer-friendly.
My demo has a module which provides a JAX-RS resource which is consumed by
a Jersey module which provides an HttpServlet which is again consumed by a
Jetty module which deploys the servlet in a servlet context.
Redeploying the module containing the JAX-RS resource takes something like
50ms IIRC.
Cheers,
Eirik.
Re: Loading an automatic module from an exploded directory
On 09/04/2020 16:42, Eirik Bjørsnøs wrote:
The current implementation of automatic modules seems to assume that an
automatic module is always packaged as a jar file.
I'm working on a module runtime where this is not always the case, and the
limitation has become a bit of a challenge.
I want to package applications (modules + runtime) at build time into a
single jar for distribution.
The runtime loads modules from directories within its own jar. This means
that ModuleFinder.of(Path..) receives module locations in the form:
jar:file:///path/to/single.jar!/META-INF/modules/module-1.0/
This works fine as long as modules are explicit. (With the unrelated
limitation that the multi-release feature also seem to assume jar files)
The only packaging format for automatic modules that Java SE defines is
JAR files. The "Multi-release JAR files" feature is also JAR file only.
If I read your mail correctly, you are creating "multi-module JAR files"
where the modules are "exploded" under /META-INF/modules in
${NAME}-${VERSION} directories. It shouldn't be too hard to create your
own ModuleFinder that finds modules under META-INF/modules. This would
mean implementing ModuleFinder rather trying to use
ModuleFinder.of(Path...). I assume you've found ModuleDescriptor.read to
read/parse the module-info.class of explicit modules. You are right that
it would require code to scan directory trees, at least the equivalent
of automatic modules, maybe for explicit modules too. However, it
shouldn't be too hard. Have you tried the zip file system provider? That
would allow you to open the JAR file as a file system so you can use the
file system API.
:
I have also identify an additional use case which is to allow hot-deploying
automatic modules during development from target/classes using a Maven
plugin.
I'm not sure how to interpret this but just to say that the unit of
replacement is the module layer, you can't replace modules in a layer
and/or dynamically change the set of packages in a loaded module.
-Alan
