Re: Is it safe to update jar that are used for dynamic layers?

[Alan Bateman]

On 06/05/2022 12:26, Alex Orlov wrote:

Hello.
  
Please, consider the following situation. We have /home/user/temp/foo.jar that is used in dynamically created jpms

layer bar.
  
Could anyone say if it is safe to do the following: destroy layer bar, update /home/user/temp/foo.jar (modify some code in this jar)and create a new jpms layer (bar_2) that will use /home/user/temp/foo.jar.


This isn't really a module system issue, it's more about whether about 
you know if a JAR file is still open (and in use) and whether you can 
replace a JAR file that is open. In general it is not safe/reliable to 
do this. If you search JBS you will find dozens of bug reports on this 
topic, most going back many years. Many of the older bug reports are 
crashes because the central directory of a zip file was historically 
memory mapped and accessing a memory mapped region after the file has 
been replaced will typically SIGBUS. You'll also see bug reports on 
Windows with errors replacing zip or JAR files that are open. For the 
module system then you could use a weak reference and have it be queued 
when the module layer is GC'ed. However, I suspect this may not be 
timely enough for what you are doing and maybe you could look at using a 
unique name/location for each update instead.


-Alan

Re: Current Project Status

[Alan Bateman]

On 03/04/2022 09:26, Julian Waters wrote:

Out of curiosity, is the Jigsaw project still active in the background
today, seeing that this mailing list is still up?

The features developed by this project were integrated in JDK 9. The 
mailing list lives on, it is used periodically for questions, bug 
reports, and related discussion.


-Alan

Re: Is it better for JDK to distribute jmod files independently of JDK itself?

[Alan Bateman]

On 08/02/2022 14:17, Glavo wrote:


:

In addition, I think there are other better options for the out-of-the-box
availability of jlink.
I noticed that a copy of everything in jmod already exists in the JDK
folder.
In fact, jmod of JDK core module only needs to store a file list (List of
paths of all native libraries, executable files and legal files),
original content backup of some modifiable content (e.g. conf files), and
the name of module.
We can re extract the complete jmod file from JDK through these meta
information, the size of these meta information is completely negligible.
I'm spending some free time trying to implement such a tool, and I don't
think it's difficult.
Just an FYI that there are transformations and some class generation at 
link time that means the classes/resources in the run-time image aren't 
identical to the those in the packaged modules.


-Alan

Re: DFS vs BFS search of modules during resolution

[Alan Bateman]

On 03/12/2021 13:39, Michał Kłeczek wrote:

Am I missing something?

I think you understand it correctly. The notion of multi-parent 
configuration and layers is something that was added for those that were 
focused on interop with other class loader based systems. It was tracked 
as issue #NonHierarchicalLayers [1].


-Alan

[1] 
https://openjdk.java.net/projects/jigsaw/spec/issues/#NonHierarchicalLayers

Re: ModuleLayer.Controller strongly references Module

[Alan Bateman]

On 28/11/2021 22:29, Michał Kłeczek wrote:

:
It looks like ML.Controller.addExport(source, pn, target) does not update the 
target module ClassLoader view of exported packages in the default ClassLoader 
implementation (jdk.internal.loader.Loader).
What is the reasoning behind such implementation? I am wondering now how useful 
Controller.addExport() really is...

The addXXX methods are about accessibility. In the case of addExports 
then maybe you want to export an internal package so that code in a 
module in the child layer can make use of (via reflection) public 
classes in that package.


You are right that the addXXX methods doesn't do any magic for 
visibility. For super advanced cases where someone is using their own 
class loaders then they need to ensure that the class loader delegation 
supports the readability graph. Same thing when they use addExports 
where they may need to their class loaders to adjust their delegation to 
support changes at run-time.


-Alan

Re: ModuleLayer.Controller strongly references Module

[Alan Bateman]

On 24/11/2021 18:14, Michał Kłeczek wrote:



For sure it’s fun :)

Regardless of “smart proxies” the lifetime of ML.Controller should be 
- if not fixed - at least be documented IMHO. Right now it’s a minefield.


It's also somewhat unusual so I think we'll need to think if this is the 
right thing to do.


The ML.Controller for the boot layer is not needed and I think it would 
be attractive nuisance to have it hanging around.


-Alan

Re: ModuleLayer.Controller strongly references Module

[Alan Bateman]

On 24/11/2021 12:19, Michał Kłeczek wrote:

Hi Alan,

(Disregard previous message - fat fingers)

Attached is a first draft of the patch:

1. Controller is created upon ML construction and returned from static 
ML.defineModules methods.
2. Created two controller liveness tests (they rely on System.gc() 
though - not sure if there are some tricks to test it reliably).


I read your mail about "smart proxy" classes and lazy downloading of 
modules. I don't have an opinion at this time on whether this is worth 
doing or not.


If you want to experiment with patches then add a volatile field to ML 
and only set it when VM.isBooted() is true. In order words, let the 
ML.Controller for the boot layer be GC'ed as it is now (it would be 
attractive nuisance to keep it alive). There are many tests in the 
test/jdk tree that check that a weak ref is cleared, there is also some 
test infrastructure to help, so you might want to look at those.


-Alan

Re: ModuleLayer.Controller strongly references Module

[Alan Bateman]

On 22/11/2021 07:23, Michał Kłeczek wrote:


Would it be too much to ask for accepting a patch adding a reference 
to ML.Controller to ModuleLayer?
It wouldn’t require changes to the spec I think as the lifetime of 
ML.Controller is unspecified.


Assuming it's not exposed, and that there is no reference to the 
ML.Controller for the boot layer, then it might be okay but I think it 
would be useful to understand the use-case first. As I assume you know, 
the ML.Controller allows containers to do the equivalent of --add-reads, 
--add-exports, and --add-opens in child layers.  Since you want to 
retain the reference to the ML.Controller around then it may be some 
other reason, maybe opening packages to descendants beyond the child layer?


-Alan

Re: Alternative to fatJar - modular solution?

[Alan Bateman]

On 11/10/2021 15:09, Glavo wrote:

I think this is a great prototype. Based on it, I think such requirements
can also be realized by enhancing jar in these aspects:

   1. Nested jar file system (The ujar file system seems unnecessary.
  I never understand why jar file systems cannot be nested.)


This was fixed in JDK 12, are you seeing issues with release recent 
releases? If so then would it be possible to submit a bug with a test 
case or bring the issue to core-libs-dev?


-Alan

Re: Alternative to fatJar - modular solution?

[Alan Bateman]

On 06/10/2021 18:24, Glavo wrote:

For a long time, unpacking and repackaging all dependencies into a file
called fatJar was the first choice for single file distribution of Java
programs. However, the compatibility of this solution with JPMS is very
poor - it breaks up all the modules and works with classpath.
Project Jigsaw did have a number items related to module artifacts that 
it didn't get to. These included a means to package several modules into 
a single artifact and executable uber-JARs, in both cases preserving 
module boundaries and identity. There were a number of prototypes and 
some initial exploration but they had to put aside to focus on the core 
platform. As Ioi shows, the APIs to do your own experiments are there so 
there is nothing stopping you, or others, from doing more exploration 
and working through the many issues and directions that might arise. 
That's all useful to do in advance of some future JEP, if someone is 
willing to invest in this area.


-Alan.

Re: Alternative to fatJar - modular solution?

[Alan Bateman]

On 09/10/2021 05:34, Samuel Audet wrote:

One problem is that frameworks like Spring Boot need to scan the classes
that are available in a module, to get a list of the names of all the
classes, among other things. The JDK provides no standard way to do that.
For example, how would you implement a class like this one?
The ModuleReader is the API for listing or accessing the contents of a 
module. With these frameworks then then it would be a lot more efficient 
to do these things at build or link time. At one point there was 
discussion and prototypes of doing the indexing in a jlink plugin.


-Alan

Re: module resource loading - opens {package} when {package} only holds resources

[Alan Bateman]

On 20/09/2021 23:15, Rob Bygrave wrote:

/:/

In case it's interesting, one thing that I hit when migrating to 
module-info was that a number of the modules are only used in maven 
test scope - as such they don't get a /requires/ clause in 
src/main/java/module-info. Currently when running tests via maven, 
surefire, junit5 there is an unnamed module created for testing 
purposes that is patched and adds the main module. Currently these 
maven test scope only modules are getting treated as unnamed when 
running tests and so at this point can't be referenced in an /exports 
to/ clause of another module. The net result is that I have some 
/exports/ clauses that ideally would be /exports to /but can't at this 
stage. There might be a detail I am missing but this is ok and 
probably isn't a common case.


I don't think I understand the issue. The compiler plugin should compile 
the tests "as if" they are part of the module. When the tests execute 
then the Surefire plugin should be do the same as compile-time and patch 
the module to include the test classes. Under the covers I would expect 
both plugins to use --patch-module, --add-reads junit5, and maybe 
--add-exports to export packages to junit5. It sounds like there may be 
a mismatch between compile-time and run-time but I can't be sure without 
seeing more.


-Alan

Re: jlink SystemModulesPlugin use of hashCode breaks reproducibility

[Alan Bateman]

On 20/09/2021 12:07, Craig Raw wrote:
Sure - I'm using the Gradle badass-jlink-plugin. The command it is 
executing is:


/Users/craigraw/.sdkman/candidates/java/16.0.1.hs-adpt/bin/jlink -v 
--strip-debug --compress 2 --no-header-files --no-man-pages 
--ignore-signing-information --exclude-files **.png 
--exclude-resources glob:/com.sparrowwallet.merged.module/META-INF/* 
--module-path 
/Users/craigraw/.sdkman/candidates/java/16.0.1.hs-adpt/jmods/:/Users/craigraw/git/sparrow/build/jlinkbase/jlinkjars 
--add-modules com.sparrowwallet.sparrow --output 
/Users/craigraw/git/sparrow/build/image


I've had no experience with JDK 17 yet - I'm specifying that users 
build with a constant version (JDK 16 at present) to maintain 
reproducibility with jpackage.


I see you've figured jimage extract so extract the classes/resources. 
Would it be possible to make available the SystemModules$default.class 
and SystemModules$all.class files from two runs of the above that 
produce non-identical class files?


-Alan

Re: module resource loading - opens {package} when {package} only holds resources

[Alan Bateman]

On 20/09/2021 11:13, Rob Bygrave wrote:
Well ok, silly me really because I think it is now clear that the 
"proper answer" is that the module really needs to open the 
subdirectories.

opens dbmigration.postgres; opens dbmigration.mysql;

So thanks and apologies Alan. It's all making sense to me now, 
unfortunately I got muddled by that error.


np, but I assume the only reason you open these packages is that you 
expect code outside of your module to locate these resources. If the 
only code that locates the resources is in your module then there should 
be no need to open them.


-Alan

Re: jlink SystemModulesPlugin use of hashCode breaks reproducibility

[Alan Bateman]

On 20/09/2021 07:52, Craig Raw wrote:

Hi,

While I've noticed significant improvements in making reproducible builds
with JDK 16, there is a remaining issue I'm stumped by. The issue is this:

Using jlink invokes the SystemModulesPlugin, which creates .class files
containing module descriptors for all the modules of the application.
Specifically, these .class files are SystemModules$all.class and
SystemModules$default.class. The class files create instances of
jdk.internal.module.Builder to create the ModuleDescriptor objects. The
ModuleDescriptors are created by use of the Builder.build(int hashCode)
method. The problem is this: the value of the hashCode parameter can change
when building across different systems, changing the bytecode of the
SystemModules$*.class files, and therefore the modules file created by
jlink.

It is unclear to me why the hashCode should differ. The SystemModulesPlugin
invokes ModuleDescriptor.hashCode() on line 1116, and that method should be
deterministic. However, I do note that the value is cached inside
ModuleDescriptor, and in fact a ModuleDescriptor can be constructed with a
provided hashCode value. Perhaps an outdated value is cached? My other
consideration is that integer overflow is behaving differently on some
systems, but I am targeting 64 bit systems only, and this issue is
occurring across similar Debian-based platforms running identical JDK
versions (adoptopenjdk-16-hotspot=16.0.1+9).

The issue is illustrated with an example here:
https://github.com/sparrowwallet/sparrow/issues/197

Note that this issue is intermittent - many users are able to create
reproducible builds of this project, but some are not. For reference, the
steps to creating a reproducible build are documented here (note only the
1.5.0-beta1 release is reproducible):
https://github.com/sparrowwallet/sparrow/blob/master/docs/reproducible.md

Any assistance in resolving this is appreciated.
We fixed a lots of issues with reproducible builds in recent releases, 
including several issues related to iteration order that were 
problematic for the jlink plugins. Off hand I can't think of any issues 
with the ModuleDescriptor hashCode. It is computed at link time and 
should be deterministic. If I were to guess then then this may be 
something to do with the module version recorded at compile-time at that 
is one of the components that the hash is based on. That's just a guess, 
I think I would need to see the module path specified to jlink to have a 
better idea.


BTW: Have you had any reports with JDK 17 yet?

-Alan

Re: module resource loading - opens {package} when {package} only holds resources

[Alan Bateman]

On 20/09/2021 09:55, Rob Bygrave wrote:

Hi,
I have a case where I believe I have a module where I would like to use:

opens {package};

... *where {package} only contains resources*, there are no .class files in
{package}. Currently when {package} only contains resources, we get an
InvalidModuleDescriptorException error:

Error occurred during initialization of boot layer
java.lang.module.FindException: Error reading module:
/home/rob/github/avaje/test-modules/target/classes
Caused by: java.lang.module.InvalidModuleDescriptorException: Package
dbmigration not found in module

module test.modules {
   ...
   opens dbmigration;
}
You can open a package that only contains resources. The directory 
structure above hints that this is a Maven project, it is possible that 
the "dbmigration" resources have not been copied into target/classes?


-Alan

Re: Modules with platform specific parts

[Alan Bateman]

On 15/09/2021 09:45, Johan Vos wrote:

Hi,

There have been discussions in the past about how to deal with
platform-specific parts (java code, native code, resources) in modules.
There is no standard for this, and afaik no recommendation. In the OpenJFX
project, we upload jars with module info to maven central, and we have
plugins for maven and gradle to deal with them at compiletime and at
runtime.

Project Panama and the foreign linker API may be another motivation to 
re-visit this topic. One general concern is adding more complexity to 
the JAR format. I could imagine platform specific sections adding 
another dimension of complexity to MR JARs, modular JARs, modular MR 
JARs, ...  Also every bag we nail on could have implications for compile 
time, packaging time, jlink, and run-time.


The position we took on this in JDK 9 is that modules can be platform 
specific. There is a JDK-specific class file attribute named 
"ModuleTarget" for the OS and/or architecture.  The jmod tool has an 
option to specify the OS/arch when creating a JMOD. Post resolution 
checks will catch issues where someone accidentally deploys a module for 
the different OS/architecture. I have not looked at the JavaFX modules 
recently but at some point I think they were packaged as JMOD files and 
it might be that the platform was specified at that time. If you've 
since moved to modular JARs then I could imagine this getting lost.


-Alan

Re: Portable retrieval of resource bundles across module path and class path

[Alan Bateman]

On 23/07/2021 18:23, Gunnar Morling wrote:

:

Yes, I'm quite sure, unless I'm doing something really stupid :) 
Here's the steps for reproducing:


git clone g...@github.com:gunnarmorling/resource-bundle-test.git
git checkout split-package
cd resource-bundle-test
mvn clean install
jar -tf german/target/resourceloading-test-german-1.0-SNAPSHOT.jar
jar --describe-module --file 
german/target/resourceloading-test-german-1.0-SNAPSHOT.jar


jar -tf shows the 
resource dev/morling/greeter/fr/GreetingMessages_de.properties, but 
jar --describe-module doesn't show the dev.morling.greeter.fr 
 
package.


I wasn't able to duplicate this and don't immediately see how how this 
project ends up with GreetingMessages_de.properties in 
dev/morling/greeter/fr.




> I think this discussion comes down to how the set of packages for an 
explicit module is determined.


Agreed, and I think I got my answers. I'll try and do a quick blog 
post about this topic, I haven't seen much discussion of 
ResourceBundleProvider for instance.


ResourceBundleProvider is an advanced feature and most projects will 
likely get by by including the resources with the application. It's only 
when you separate them and wish to deploy the translations as modules 
that you need to know about this feature.


-Alan

Re: Portable retrieval of resource bundles across module path and class path

[Alan Bateman]

On 23/07/2021 16:58, Gunnar Morling wrote:

:


Yes, there is such resource which I had created for demo purposes (see 
the jar -tf  output above):


dev/morling/greeter/fr/GreetingMessages_de.properties

Here's the output you requested:

jar --describe-module --file 
german/target/resourceloading-test-german-1.0-SNAPSHOT.jar
dev.morling.greeter.german@1.0-SNAPSHOT 
jar:file:///.../resource-loading-test/german/target/resourceloading-test-german-1.0-SNAPSHOT.jar!/module-info.class

requires dev.morling.greeter.base
requires java.base mandated
provides dev.morling.greeter.spi.GreetingMessagesProvider with 
dev.morling.greeter.de.internal.GermanGreetingMessagesProvider

contains dev.morling.greeter.de.internal

Interestingly, the dev.morling.greeter.fr 
 
package of the GreetingMessages_de.properties resource file isn't 
listed there, but it does trigger the split package error later on. 
Based on what you say, I take it that split package error *is* 
expected in this situation, it' surprising though to not see the 
package listed in the output of --describe-module.


If you wanted to reproduce this yourself, the code is here:

https://github.com/gunnarmorling/resource-bundle-test/tree/split-package 



Just run "mvn clean install" on JDK 9+, then "mvn exec:exec -pl 
:resourceloading-test-runner" for running the application, which 
should fail.


Are you sure this always puts 
dev/morling/greeter/fr/GreetingMessages_de.properties into 
resourceloading-test-german-1.0-SNAPSHOT.jar? I'm quite sure the JAR 
file above doesn't have the fr resource but the JAR file in your 
previous mail seems to include it.


In any case I think this discussion comes down to how the set of 
packages for an explicit module is determined. As explicit modules 
encapsulate their resources by default then it's the set of package 
derived from the non-directory entries that map to legal package names. 
This may be determined at packaging time by the "jar" tool that records 
the set of the packages in the ModulePackages class file attribute, or 
at run-time by scanning the contents of the JAR file. (We need to do a 
better job of specifying that [1]).  So it's different to automatic 
modules where they is no encapsulation and only the .class files are 
used to determine the set of packages.


-Alan

[1] https://bugs.openjdk.java.net/browse/JDK-8230298

Re: Portable retrieval of resource bundles across module path and class path

[Alan Bateman]

On 23/07/2021 12:17, Gunnar Morling wrote:

:

> For the migration scenario where the resources are in .properties 
format then the simplest may be to just deploy the JAR files on the 
module path where they will be treated as automatic modules.


In the scenario I have in mind (for educational purposes mainly) the 
JAR files contain the properties file as well code and should be 
usable as "proper" (i.e. non-automatic) modules as well as on the 
classpath.


> There are no split package issues unless the resources have been 
compiled as .class files.


That's very interesting, it's not what I observe. Having the 
*.properties in one and the same package in multiple modules triggers 
an error upon start-up:


I should have been clearer that the "no split package issues" comment 
was in the context of automatic modules where you "move" an existing JAR 
file containing .properties files from the class path to the module 
path. The .class files in the JAR file are used to determine the set of 
packages in the automatic module so if there are no .class files then it 
doesn't have any packages.


That said, I'm surprised by the exception message as suggests there must 
be dev.morling.greeter.fr classes or resources  in the 
dev.morling.greeter.german module. I think I'd need to see the output of 
`jar --file= --describe-module` to be sure.


-Alan

Re: Portable retrieval of resource bundles across module path and class path

[Alan Bateman]

On 22/07/2021 23:30, Mandy Chung wrote:


I assume the class path is running on JDK <= 8, right? Otherwise

This is a reasonable approach for the resource bundles to load from 
both class path and module path when the resource bundles of a given 
name is packaged in multiple modules/JARs.


We considered extending the default ResourceBundle.Control 
implementation to search with a repackaged bundle name as 
AbstractResourceBundleProvider::toBundleName in addition to the 
current default Control::toBundleName.  This incurs higher lookup cost 
(e.g. when it fails with the search of p.foo.Resource_, it will 
search p.foo..Resource.).  What you have is reasonable.


Mandy
I agree. Also just to point out that the ResourceBundle javadoc has a 
"Resource Bundles and Named Modules" section to outline the options for 
deployed resource bundles as modules. For the migration scenario where 
the resources are in .properties format then the simplest may be to just 
deploy the JAR files on the module path where they will be treated as 
automatic modules.  There are no split package issues unless the 
resources have been compiled as .class files.


-Alan

Re: Requires runtime considered?

[Alan Bateman]

On 28/06/2021 09:49, Christian Beikov wrote:

Hello experts,

I am in the process of modularizing one of my projects and hit some 
rather annoying limitations that requires me to change parts of my 
build and dependencies due to the way the Java module system works.


I usually have modules like x-api, x-impl and x-testsuite where I have 
a compile time dependency from x-impl to x-api. In x-testsuite I only 
code against x-api, so I define a Maven runtime dependency on x-impl 
in order for my CDI container to pick up the implementations. The 
annoying part is that with modularization I now need to declare my 
dependency in x-testsuite on x-impl as compile time dependency, 
because otherwise I can't list the module in the module-info. If I 
don't list the module, then it is not "visible" to the CDI container, 
which I guess is on purpose, even though it probably ends up in the 
module path.


Is that by design that classes are not visible if no module depends on 
a module like x-impl in such a scenario, or is that maybe something 
that could work and should be improved in the CDI implementation? I 
was thinking that if "requires runtime ..." were allowed without 
checking that the module is available during compilation, this would 
work fine. Did this idea come up before already?


The module system supports services. x-api uses a service, x-impl 
provides an implementation of the service. If x-api is resolved then all 
observable modules that provide an implementation of the services used 
by x-api will be resolved too. In this scenario, x-impl requires x-api, 
x-api does not require x-impl. You used the phrase "compile time 
dependency" but it's more than that, it's all phases as there will be 
references in x-impl to x-api's exported API.


I don't know if you control the CDI implementation or not but if you 
aren't using services then you'll need to help the module system with 
"--add-modules x-impl". This is because no other module requires x-impl 
and it's not providing an implementation of a service. It may be 
observable but it's the same directory as 99,999 other modules that you 
don't care about.


-Alan

Re: Module resource loading

[Alan Bateman]

On 08/06/2021 20:20, Alexey Gavrilov wrote:

Note that *Module 2* cannot contain its own resources under `dir1` and
`dir2` directories because they are already encapsulated in *Module 1*. If
you try adding `dir1` you will get the following error:

 Error occurred during initialization of boot layer
 java.lang.LayerInstantiationException: Package dir1 in both module
module_one and module module_two

Resource encapsulation is complicated.

The basic guideline is that code should use Class::getResourceXXX or 
Module::getResourceXXX to locate a resource in its own module. 
ClassLoader::getResource is for locating other resources in other 
components on the class path or module path. Services are often a better 
choice for cases where the resource is configuration that names a class.


Two modules with resources in the same "package" is the same as split 
package issue. Are the two modules in question closely related, meaning 
sharing a name space, or they are completely unrelated?


-Alan.

Re: Fwd: Add reads to open module

[Alan Bateman]

On 17/05/2021 16:43, Christian Beikov wrote:

Hi,

I have a use case where I generate some code and that code refers to 
classes of some module B.


When the module A (an open module), into which I define that class, 
has no read-edges to the module B, this fails saying that A does not 
read B.


That's fine and totally understandable, so I tried to add the 
following code:


moduleA.addReads("moduleB.pkg", moduleB)

but this is not allowed as `addReads` only works if `this == 
callerModule`.


Have you tried generating this in the  of the class that you 
inject? That should be a lot simpler than inject another class or 
complicating the existing API.


-Alan




My workaround is to define a class into module A that can then call 
this method.


Is there a reason to this limitation? I mean the module is open 
anyway, so anyone can just define a class into it, so why not also 
allow calling addReads?

Re: Opening -javaagent classes to named modules

[Alan Bateman]

On 11/05/2021 16:10, Fabian Meumertzheim wrote:

I am currently working on a JVM fuzzer (
https://github.com/CodeIntelligenceTesting/jazzer/) and would like to apply
it also to classes delivered with the JDK, e.g. the image parsers in
javax.imageio.* in the java.desktop module. The fuzzer uses a Java agent to
add coverage instrumentation (similar to JaCoCo) to classes at runtime.

As opposed to JaCoCo, for technical reasons the fuzzer needs to inject
bytecode referencing a public static byte[] field defined on a class of the
agent. Since -javaagent does not support Java modules, this class will be
part of the unnamed module and thus can't be "opened" to other modules,
leading to a "NoClassDefFoundError" when the bytecode is injected into a
class in a different module, e.g. java.desktop.

Is there any way to have a Java agent open up a class for direct access by
named modules? I have full control over the JVM args within the fuzzer, but
haven't found a way to achieve this.
I don't think this a modules issue, it's more likely a visibility issue 
because the boot class loader does not delegate to the application class 
loader. If you are instrumenting classes in java.desktop to call into 
some supporting classes that the agent provides then you need those 
class on the boot class path. 
Instrumentation::appendToBootstrapClassLoaderSearch is the API for that. 
You'll also need to use redefineModule to change java.desktop to read 
the unnamed module of the boot class loader.


-Alan

Re: Difference of module jdk.xml.dom and module jdk.zipfs

[Alan Bateman]

On 26/03/2021 10:09, Christian Stein wrote:

Thank you for the clarification, Alan.

First, I fixed the problem as you suggested by adding
--add-modules ALL-SYSTEM to the java command.
It yielded warnings about incubator modules, though:

  WARNING: Using incubator modules:
  jdk.incubator.vector, jdk.incubator.foreign
I should have mentioned that, this arises because "ALL-SYSTEM" is all 
modules in the runtime image so it includes the incubating modules that 
aren't resolved by default.





Next, I went with --add-modules ALL-DEFAULT
That fixed the runtime issues with jdk.xml.dom

Later that day, Jorn Vernee suggested another
fix [1], that works (for Bach) also at runtime without
using the --add-modules X at the command line.
Basically, it puts a ModuleFinder.ofSystem() as
the after component of the custom module layer
Bach creates for running ToolProvider services.

That probably works in this case but it won't work in general because 
the standard (java.*) modules and many of the JDK-specific (jdk.*) 
modules cannot be mapped to class loader others than the boot and 
platform class loaders.


-Alan

Re: Difference of module jdk.xml.dom and module jdk.zipfs

[Alan Bateman]

On 25/03/2021 14:59, Christian Stein wrote:

Hi,

Besides their names, contents and purposes. (-:

So, what's the difference of module jdk.xml.dom and
module jdk.zipfs in respect to their visibility(?) at
compile and run-time?

Suppose, there's a module test.base declared as:

   module test.base {
  requires jdk.xml.dom;
  requires jdk.zipfs;
   }

Compiling it with javac (from JDK 16) works like a charm.
Running it in a custom module layer fails with:

   java.lang.module.FindException:
 Module jdk.xml.dom not found, required by test.base

If I comment out the "requires jdk.xml.dom;" directive,
everything is fine. Same, if I start the test running process
with "--add-modules DEFAULT" as an argument.

Am I setting up the module layer in a wrong manner? [1]
Are there differences between system modules?
Like all = system 1 + system 2 + incubating?
jdk.zipfs is a service provider module. It `provides` an implementation 
of j.nio.spi.FileSystemProvider. The java.base module `uses` this 
service so this is the reason why jdk.zipfs is in the boot layer. 
Running with --show-module-resolution is a useful way to understand why 
a module is in the boot layer.


jdk.xml.dom exports org.w3c.dom APIs that are not defined by Java SE. 
It's not in the boot layer in your test because none of the modules 
requires it.


When you attempt to create the configuration for a module layer at 
runtime then you are (I assume) using the configuration for the boot 
layer as the parent. The jdk.xml.dom module is not observable with the 
module finder you are using and it's not in the parent configuration so 
this is the reason why you get the FindException "module jdk.xml.dom not 
found" exception.


There isn't any way to add to the set of modules in the boot layer at 
runtime so you have the help the module system by running with 
-add-modules jdk.xml.dom. More generally, a container-like program that 
is loading plugins and creating module layers at runtime will probably 
want to run with --add-modules ALL-SYSTEM to ensure that all "system" 
modules are in the boot layer and avoid disappointment at runtime.


-Alan

Re: Add information of which automatic module doesn't the

[Alan Bateman]

On 25/03/2021 14:11, Christian Stein wrote:


On Wed, Mar 3, 2021 at 10:10 AM Alan Bateman <mailto:alan.bate...@oracle.com>> wrote:


[...]
Thanks, it should include the file path to the JAR file in the
message
to make it easy to find the JAR file that can't be used as an
automatic
module. I'll create an issue to track that.


Did you create an issue, Alan? If not, I'll create one and give it a shot.


Yes, JDK-8262944 [1], I should have replied back at the time with the link.

-Alan.

[1] https://bugs.openjdk.java.net/browse/JDK-8262944

Re: New candidate JEP: 403: Strongly Encapsulate JDK Internals

[Alan Bateman]

On 25/03/2021 10:54, Andrew Haley wrote:

:
Looking at the JEP, I don't immediately see anything about long-term
use of illegal accesses in test code. There are a good many cases where
we need to test private and protected classes and fields, for example.
Is there a proposal to allow this?

I don't think tests should be impacted by this JEP.

For the JDK tests, jtreg has the @modules tag to do what we want. It can 
be used to export and/or open any of the packages for the test, the 
value of the tags get translated to the equivalent of --add-exports or 
--add-opens.


More generally, tests for modules, say white box tests that exercise 
package private methods, can continue to be compiled and run "as if" 
they are part of the module. The Maven/other plugins for testing will 
use --patch-module, and maybe other options, to the setup for the tests.


-Alan.

Re: Add information of which automatic module doesn't the

[Alan Bateman]

On 03/03/2021 01:32, Thiago Henrique Hupner wrote:

When running some old applications in the module path, sometimes a jar
provides a service that doesn't exist in it.
So, the module system correctly informs about it, however, it doesn't
specify which module has this issue, so the way of debugging is testing
each jar individually with the --list-modules.

I think would be great if it was possible to change the exception in the
jdk.internal.module.ModulePath to show the module that has this problem.

Currently, the message is the following:
String msg = "Provider class " + cn + " not in module";
throw new InvalidModuleDescriptorException(msg);

Thanks, it should include the file path to the JAR file in the message 
to make it easy to find the JAR file that can't be used as an automatic 
module. I'll create an issue to track that.


-Alan

Re: Is RuntimePermission.accessClassInPackage now redundant?

[Alan Bateman]

On 15/02/2021 19:45, Mike Hearn wrote:

Hello,

I was reviewing the default security policy and noticed quite a few modules
are given accessClassInPackage permissions to e.g. packages in the sun.*
hierarchy. Are the SecurityManager/ClassLoader based package control still
needed now the module system has a built-in equivalent? I wasn't fully sure
if this is some sort of tech debt waiting to be cleaned up, or if these
permissions still have a role to play in the post-Jigsaw world?
This is a good observation. There was a lengthy discussion about this 
here in 2017. The summary is that we want the package access check to go 
away. It will mean a few subtle behavioral changes. In particular, 
IllegalAccessError may be thrown for cases where SecurityException is 
thrown today (assume running with a SM). It may also change the point 
where an exception is thrown due to verification triggering class 
loading and a package access check.


-Alan

Re: jlink images and module layers

[Alan Bateman]

On 10/02/2021 21:44, Gunnar Morling wrote:

Hi Alan,

Thanks for your thoughtful reply. The use case I'd see for layers
within jlink images is isolation of different (transitive) dependency
versions within an application. While this isn't present (and I
understand it'd add a fair share of complexity), we can work around
this by putting the modules of different layers into a separate
directory and use this as source for the layers running on top of the
JDK modules from the image. We might envision some tooling support for
creating such "extended" jlink image.
It would need a lot of consideration and exploration before going there. 
One reason for having different versions of a module in the run-time 
image might be to support different initial modules, say a run-time 
image containing two distinct applications or tools. Both would run with 
in flat name space but for whatever reason can't use the same version of 
some module. At the other extreme might be launching an application that 
is magically started with several layers and different versions of 
modules in each layer. Somewhere in the middle is a container in the 
boot layer and at run-time it uses the APIs to create module layers. The 
set of observable modules that it uses when creating the configurations 
for these modules layer is some subset of the modules in the run-time 
image that include modules that weren't observable at startup when 
creating the boot layer.






Btw. when exploring dynamic plug-in layers and their interaction with
services, we noticed that you'll also get services provided by parent
layers. This leads to code patterns like shown here:

 
https://urldefense.com/v3/__https://github.com/moditect/layrry-examples/blob/master/modular-tiles/core/modular-tiles-core/src/main/java/org/kordamp/tiles/core/TilePluginLifecycleListener.java*L41-L43__;Iw!!GqivPVa7Brio!L0YyyA2bQu9Mr2QdbMC3mK-U7Vkt8WmI3LIlw_FmwS58Iqo42j9JN77ETD7lxI-Q-Q$

I.e. you need to filter out all those service implementations from
parent layers. It would be nice if there was a flavour of load()
(perhaps just a boolean flag "includeParentLayers"), which would make
this a bit simpler. Also, IIUC, the code using the service loader
needs to have awareness of the fact that it runs within layers, as the
traditional load() methods wouldn't find any implementations from
parent layers, which might present a migration obstacle when moving
existing libraries to layered architectures.
The load(ModuleLayer, service) is specified to load service providers in 
the given layer and its ancestors. That is consistent with the load 
method that takes a ClassLoader as that has always been specified to use 
the class loaders that are reachable via parent delegation.


The stream() method might be useful for the example as you could use a 
filter to selects providers in the desired layer.


What parent class loader do you specify to defineModules when you 
are creating the module layers? I'm asking because the traditional 
ServiceLoader.load will locate providers in named modules when the class 
loader specified to the method has modules in a module layer. This goes 
for class loaders that are reachable via parent delegation too. When 
Layrry creates a layer with defineModules then it must specify the 
parent class loader for the resulting class loaders. If you choose one 
of the class loaders from the layer parent (any of them will do) then it 
might get more of the traditional usages of ServiceLoader.load working.


-Alan

Re: jlink images and module layers

[Alan Bateman]

On 05/02/2021 15:39, Gunnar Morling wrote:

Hi all,

While working on Layrry [1] [2] [3], a launcher and runtime for
layered Java applications, I've come to wonder whether there's a way
for sourcing multiple module layers from a custom runtime image
created via jlink?

I couldn't find a way so far, as a) it seems not possible to add
multiple versions of a module to a runtime image (one of the use cases
for layered applications), and b) it's not clear how the module finder
API would address modules in an image. If this is not supported atm.,
are there any plans to do so?

To give some context on Layrry, it's an exploration for providing a
framework that allows the creation of modularized applications based
on the module system and its notion of layers, addressing the
multi-version issue of (transitive) dependences, provisioning modules
based on their Maven coordinates, supporting the dynamic
addition/removal of layers at runtime ("plug-ins"), and more.


No, there isn't any support for this.

As things stand, the universe of observable modules at startup includes 
all modules in the run-time image. If a run-time image were to include 
modules only intended for module layers created at run-time then there 
would need to be configuration or something to support the partitioning, 
also maybe APIs to expose enough at run-time to reconstitute the 
configuration. There would be restrictions on the content of these 
modules, e.g. modules intended only for child layers couldn't contribute 
command-line launchers for the bin directory. The container format used 
for classes and resources would need to be rev'ed to support more 
complex addressing (this is something we explored a bit in JDK 9 but 
didn't go too far), or the run-time image structure would need to be 
updated to support directories of observable modules for each 
configuration. A lot of complexity would be heaped on jlink as it would 
have to create the configuration for each of the layers at link time, 
maybe even a training run to map the modules to class loaders to 
identity consistency issues at link-time. Overall I think it would add a 
lot of complexity in many areas.


Module layers are great for containers that launch applications that use 
different versions of components, it's also great for plugins and 
dynamic behavior (the "Finding Plug-In Implementations ..." section in 
your blog is good). I'm less sure about trying to bring it to the link 
phase where it's about static configuration and link-time optimizations.


-Alan

Re: IllegalAccessError with --patch-module

[Alan Bateman]

On 07/01/2021 13:06, Johannes Kuhn wrote:

:

To be honest - I don't quite understand the bug - but I was able to 
reproduce it.
So I did manually retrace the code to see where the difference was 
happening, and found the lines above - which was "good enough" for an 
explanation.


Resolution involves recursively enumerating the "requires" directives of 
a set of root modules (think transitive closure). The root modules are 
the module you specify to -m or the modules specified to --add-modules. 
Every module requires java.base so it will always be resolved and will 
also be in the the boot layer. The bug in the patcher code results in a 
root module that does not require java.base and since they are no other 
root modules, it means the boot layer doesn't have java.base, oops! But, 
the java.base has been partly defined to the VM in very early startup 
and normally the creation of the boot layer will complete the 
initialization, including setting up exports. The IllegalAccessError 
trying to link the super class arises because the exports aren't setup, 
meaning java.lang is not exported so no classes outside of java.base can 
be linked.




Is there a way to debug the early java initialization? I didn't got 
that working, so had to do it by hand.
The debugger needs the VM to be fully initialized before it can debug 
Java code. In JVM TI speak this is the "live phase". There are early 
phases where tooling and agents can get events and do some operations 
during VM startup but it's not enough for a Java debugger.


-Alan.

Re: IllegalAccessError with --patch-module

[Alan Bateman]

On 06/01/2021 23:49, Johannes Kuhn wrote:

Indeed a bug.

Happens under the following conditions:
* Patched module is an automatic module
* Patch adds one or more additional packages to the module.

:

Proposed fix:
* Change requires to not check if it is an automatic module if the 
builder is not strict.

* Move the copying of the requires out of the if.


Thanks for the mails, it is bug. The additional information that the 
patch adds packages to the automatic module was important to understand 
the mails. One other thing is that the bug does not surface when the set 
of root modules includes an explicit module or an unpatched automatic 
module. So very obscure scenario that probably explains why there hasn't 
been reported before now.


The patcher should be using strict mode when re-building the module 
descriptor for a patched automatic modules. The non-strict mode is for 
explicit modules as the identifiers (that may not be valid in the Java 
Language) are read from the module-info.class. I'll create a bug for this.


-Alan.

Re: IllegalAccessError with --patch-module

[Alan Bateman]

On 06/01/2021 19:57, Thiago Henrique Hupner wrote:

Sorry, they are automatic modules.

I'll create a test case.

Also if you can include the output with --show-module-resolution then it 
might help track this down quickly (running without any explicit modules 
on the module path is, on the surface, a bit unusual but I wouldn't 
expect any issues).


-Alan

Re: IllegalAccessError with --patch-module

[Alan Bateman]

On 06/01/2021 19:15, Thiago Henrique Hupner wrote:

Hi!

I've noticed something strange, but I don't know if it is a bug:
If we have two jars:
module.a with the class com.foo.Bar
module.b with the classes com.foo.Bar and com.foo.Main.

Running:
java -p module-a.jar --patch-module=module.a=module-b.jar -m
module.a/com.foo.Main

it throws:
Error: Unable to load main class com.foo.Main in module module.a
 java.lang.IllegalAccessError: superclass access check failed: class
com.foo.Main (in module module.a) cannot access class java.lang.Object (in
module java.base) because module java.base does not export java.lang to
module module.a

Even with --add-exports=java.base/java.lang=module.a it doesn't work.

Is this a bug? I have a real usage where the main class is the "patched"
module to override the com.foo.Bar.

I'm not aware of any issues in this area. Can you create a test case as 
that might be easier to discuss? (I'm guessing the module names aren't 
"module.a" and "module.b" as that won't compile. Also I would expect 
this type of patching to emit a warning that the module-info.class in 
module-b.jar is ignored).


-Alan

Re: Should ClassLoader::getResouces return the same resource twice?

[Alan Bateman]

On 03/01/2021 13:10, Thiago Henrique Hupner wrote:

:

I hope made it a little more clear. By now it is clear that it is loading
the same
resource twice because the same jar is in the classloader and in the module
layer
classloader.
If I read your mail correctly you've got a class path today and you want 
to "move" the well behaved components to a module path, leaving the 
remaining the components that can't work as automatic modules on the 
class path. It's equivalent to specifying both a module path and class 
path to the java command line launcher. You also mention that you have 
to use a custom class loader. If that class loader were updated to 
support modules and load classes into its unnamed module then you should 
be able to get this to work. This is very advanced territory and would 
be approximately equivalent to the application class loader in the JDK 
where it supports both modules and the class path. I don't think your 
current approach of using a URLClassLoader as a parent will work as it 
doesn't support the bidirectional delegation that would be need to get 
this to work.


-Alan.

Re: Making jlink Plug-in API public

[Alan Bateman]

On 29/12/2020 19:29, Gunnar Morling wrote:

:

That's interesting; what is missing from your PoV to make the API an 
incubating one?
Getting consensus that it's the right thing to do, and then the 
commitment to doing it. The latter requires re-examining the internal 
plugin API, looking at the issues that have come up since JDK 9, going 
over each of the API elements, iteration, and then setting it on a 
course to incubate for a release or two, gathering feedback, then 
eventually make it permanent and maintaining it. On one hard, it could 
be interesting to have an eco system of useful link-time plugins.. On 
the other hand, I think many of the plugins will be tied to core modules 
so cannot be maintained outside of the jdk repo. Also many of the 
plugins that we have experience with to date involve generating classes 
or modifying existing class files at link time so there may be an 
argument to wait until the JDK has a bytecode API.




:

I could see myself contributing to a plug-in around signature 
validation, depending on how involved that task would be. I did a 
quick PoC and discuss it here:


https://www.morling.dev/blog/jlinks-missing-link-api-signature-validation/ 



This is using the existing Animal Sniffer tool for a basic signature 
check. I'm not sure how much effort it'd be to fully implement this, 
and also catch things like added non-virtual methods in a base 
class/interface, changes to type parameters, and more. Could existing 
code from the JDK be re-used for that task?
Nice blog and PoC! Yes, I think this is worth exploring more as static 
analysis to detect mismatch with compile-time could be useful. A 
possible starting point is something that scans a module path, it 
doesn't have to start out as a jlink plugin. I could imagine `java 
--validate-modules` making use of this to do deep analysis for example.


-Alan

Re: Should ClassLoader::getResouces return the same resource twice?

[Alan Bateman]

On 02/01/2021 12:59, Thiago Henrique Hupner wrote:

I guess a little context can make more things clear:
The servlet spec requires that all jars from WEB-INF/lib
be available to the same classloader.
The resource, in particular, is "META-INF/web-fragment.xml"
Each jar can contain its own. So, using getResources make sense
in order of parsing each. However, what is happening is if I have two JARs
each with its own META-INF/web-fragment.xml, using the ModuleReader
it is returning four resources, so it parses more than it should and 
it fails

to parse the same resource twice.

Which parent class loader are you specifying to 
defineModulesWithOneLoader? It it possible that it locates the resource 
in the two JAR files?


-Alan

Re: Should ClassLoader::getResouces return the same resource twice?

[Alan Bateman]

On 02/01/2021 03:21, Thiago Henrique Hupner wrote:

:

I've created a simple example of what is occurring [1].
I know there are behavior specific for getting a class if it is in a module,
but I don't know if this may be a bug in the resource loading mechanism.
In the example, the returned values are different to illustrate, but in my
case, it
returns two exact URLs for the same resource as the source for the module
reader
and the classloader is the same.
The behavior you observe with the example is correct. 
ClassLoader.getResources locates the resource by searching the lass 
loader delegation chain and in the example there are two "foo" 
resources. The resource in module fake.module is located because it is 
an automatic module that opens all its packages unconditionally. The 
second resource is located by searching the parent class loader, a 
URLClassLoader in the example that also locates "foo".  Which "foo" did 
you expect to locate? If code in fake.module just wants to locate the 
resource in its own module then it should use getResource rather than 
getResources.


-Alan.

Re: Why is it allowed to have several modules with the same name?

[Alan Bateman]

On 27/12/2020 09:50, Gunnar Morling wrote:
Thanks for clarifying, Alan. So it is intended and well-documented, 
it's still not quite clear to me though *why* that is. Wouldn#t it be 
more in line with the module system's goal of reliability to reject 
such error-prone configuration?
Reliable configuration is mostly about ensuring that there aren't 
missing dependencies, cycles between dependencies, or split packages. 
You get reliability with the module path by only reading the first 
definition of a module. If someone put 10 versions of the same module on 
the module path then classes will only be loaded from the first module, 
unlike the class path where you risk loading classes from more than one 
of the artifacts.


If it helps, the java --validate-modules option is useful for detecting 
and reporting issues. It will print an informational message when there 
are multiple definitions of a module on the module path.


-Alan

Re: Making jlink Plug-in API public

[Alan Bateman]

On 25/12/2020 10:29, Gunnar Morling wrote:

Hi all,

Are there any plans for making the jlink Plug-in API public any time soon,
perhaps for JDK 17?

I think the ability to implement custom plug-ins that are run at linking
time would open up quite a few interesting opportunities, e.g.

- removing unused members and obfuscation
- adding resources like annotation indexes for the whole image
- verifying that exported and required API methods match (avoiding
NoSuchMethodErrors due to adding incompatible module versions to an image)

The latter would be interesting in particular for monolithic applications
built jointly by multiple teams, avoiding the need to recompile the entire
application if only a single module changed, while still having fidelity
that APIs would match.

The intention in early revisions of JEP 282 was to expose a plugin API. 
It went through many iterations and re-designs during JDK 9 but it 
didn't get to the point where it was really ready to be exposed by an 
incubator module. There was also a complication at the time with 
incubator modules providing implementations of ToolProvider (that issue 
has since been resolved as it became a blocker for jpackage too).


Is it worth giving it a priority boost and re-examine it now? Hard to 
say as it would likely require several iterations and a lot of review 
cycles to bring this to incubation stage. At the same time, our 
experience with plugins to date is that many of the plugins are deeply 
tied to core modules and just aren't maintainable outside of the jdk 
repo. I think it could also be argued that some of the suggestions that 
you list should be be plugins included in the jdk.jlink module. For 
example indexing of annotations was something that was explored and 
prototyped at one point (the original requirements document for the 
module system JSR had an item along these lines). A tool that scans a 
module path to do static analysis and detect incompatibilities could be 
valuable as a link time plugin. Are these plugins that you would be 
interested in developing, contributing, and maintaining?


-Alan.

Re: Why is it allowed to have several modules with the same name?

[Alan Bateman]

On 26/12/2020 16:26, Jonathan Gibbons wrote:
If the two modules with the same name are in different positions on 
the module path, the first one will be seen and will completely hide 
any subsequent occurrences of modules with the same name.


If two modules with the same name are found in the same directory on 
the module path, such as in differently-named jar files, then that is 
an error, because neither can be said to have precedence and hide the 
other.


Just to add to Jon's mail, there detail in in the "Module paths" section 
of JEP 261 [1] and the API docs of ModuleFinder.of(Path...) [2] which is 
the API for creating the equivalent of a module path to find modules.


-Alan.

[1] https://openjdk.java.net/jeps/261#Module-paths
[2] 
https://docs.oracle.com/en/java/javase/15/docs/api/java.base/java/lang/module/ModuleFinder.html#of(java.nio.file.Path...)

Re: -Xdoclint unrecognized in JDK16 due to new module

[Alan Bateman]

On 15/12/2020 22:19, Jonathan Gibbons wrote:


On 12/15/20 12:26 AM, Alan Bateman wrote:
I think the Plexus Classworlds launcher sets a funky TCCL that 
impacts the ServiceLoader lookup of DocLint providers. Can the 
ServiceLoader usage in DocLint be changed to use the system class 
loader rather than the TCCL to see if that "fixes it"?


-Alan.



Filed as https://bugs.openjdk.java.net/browse/JDK-8258443
Okay, but you already have JDK-8253996. In that bug report, the building 
of zookeeper-jute seems to run the compiler plugin with TCCL set to 
something setup by Plexus Classworlds and this is why the DocLint 
provider in jdk.javadoc is not located at run-time.


-Alan

Re: -Xdoclint unrecognized in JDK16 due to new module

[Alan Bateman]

On 14/12/2020 20:05, Benjamin Marwell wrote:

Hello all,

it caught my attention that doclint functionality was moved to the
jdk.javadoc module [1].

Now, when calling ToolProvider.getSystemJavaCompiler().getTask(…), the
presence of the option '-Xdoclint:-missing' and similar options will throw
this exception [2]:
java.lang.IllegalArgumentException: error: invalid flag:
-Xdoclint:-missing.

This call is being used in Apache Maven, and the Apache Zookeper project
already suffers from this issue and reported it upstream [3].

Adding the modules to the maven script 'mvn' did not help. But when calling
javac as an external tool (i.e. not via ToolProvider in the same process),
the options are being recognized.

I wonder how the doclint options can be made available to
the ToolProvider.getSystemJavaCompiler().getTask(…) call again.
Maven itself (and in this case: the compiler plugin) is, as you know, a
non-modular project.
I think the Plexus Classworlds launcher sets a funky TCCL that impacts 
the ServiceLoader lookup of DocLint providers. Can the ServiceLoader 
usage in DocLint be changed to use the system class loader rather than 
the TCCL to see if that "fixes it"?


-Alan.

Re: Cannot use a mapper function to get the Platform classloader

[Alan Bateman]

On 12/12/2020 15:05, Thiago Henrique Hupner wrote:

Hi.

I'm playing around with layers and I needed the following:
Create a new layer where it doesn't have access to the boot layer using the
ModuleLayer.empty().
However, I'm getting the exception that java.base is not found. So I used
something like this:

ModuleLayer.boot().defineModules(conf, moduleName ->
myNewLayer.find(moduleName).isPresent() ? myClassLoader :
ModuleFinder.ofSystem().find(moduleName).isPresent() ?
ClassLoader.getPlatformClassLoader() : null).

Then I get the following exception: java.lang.LayerInstantiationException:
loader can't be 'null' or the platform class loader.

Why is there this restriction? How is the correct way of only enabling the
base modules to be found to the next layer and not the unnamed module.
java.base is in the boot layer and any layer you create will ultimately 
have the boot layer as its parent. If I read your mail correctly then it 
sounds like you want to restrict modules so that the only standard 
module that they read is java.base, is that correct? In that case, the 
simplest is to run with `--limit-modules java.base` so that the only 
module in the boot layer is java.base. Alternatively, maybe you should 
start with a minimum configuration like this:


    Configuration minConfiguration = Configuration.empty().
    resolve(ModuleFinder.ofSystem(), ModuleFinder.of(), 
Set.of("java.base"));


and use that as the parent configuration. If this minimum configuration 
works as the parent configuration, meaning resolve or resolveAndBind 
succeeds, then you can be assured that there are aren't any observable 
modules that require other standard modules. A scan of the observable 
modules to examine their dependences will also suffice. Then just 
ModuleLayer.boot().configuration() as the parent when creating the 
configuration for the child parent. Does that help?


-Alan

Re: Module.addOpens should log if the package has been opened for illegal access for the caller

[Alan Bateman]

On 10/12/2020 10:32, Johannes Kuhn wrote:

:

If a module has been reflectively opened, then no warning should 
emitted, right.
But reflectively opening a package should emit such a warning if you 
can only do that because it has been opened to you for illegal access.


If --add-opens is used to open the package then there shouldn't be a 
warning. The reflective case may need more consideration. None of the 
standard or JDK modules should be doing this so there may be a case to 
not suppress the warning after VM initialization has completed. In other 
words, the addOpens in your example would be a no-op because java.net is 
already open for illegal reflection access.


-Alan

Re: Module.addOpens should log if the package has been opened for illegal access for the caller

[Alan Bateman]

On 09/12/2020 22:45, Johannes Kuhn wrote:

Hope I got the right mailing list, otherwise please direct me there.

Module.addOpens currently doesn't produce an illegal access warning if 
the target package has only be opened for illegal access.


The Module methods were not intended to emit warnings. The warnings are 
supposed to be emitted by calls to AccessibleObject methods that will 
fail when the default switches to deny illegal reflective access by default.


I think your argument is that if any of the standard or JDK modules 
opens a package for illegal reflective access then code on the class 
path can use Module.addOpens to open the package, which has the 
effective of suppressing warnings. That behavior is there to support the 
--add-opens command line option.


So I think this needs a bit of thinking time to see if it's worth trying 
to do anything here. The default has switched for Java 16 and the 
intention is that the --illegal-access option will be removed by a 
future JEP. I agree it would be sad to see hacks like this in 
production, it would be far better to channel the energy being put into 
hacking around warnings to fixing fragile code to not rely on JDK internals.


-Alan

Re: Find service in a layer with multiple classloaders[2]

[Alan Bateman]

On 04/12/2020 14:25, Alex Orlov wrote:

Sorry, there was a typo in the previous message.
  
Correct > As I understand to find * FooService * in this layer we can use class loader of ANY module in this layer (any of a,b,c)...


Yes, you can specify any of the class loaders to load(service, 
ClassLoader) or just specify the module layer to load(ModuleLayer, 
service). The javadoc for both methods has all the details and hopefully 
it is clear.


-Alan

Re: MethodHandles.publicLookup().in() returns Lookup with no permissions?

[Alan Bateman]

On 24/11/2020 18:21, Simone Bordet wrote:

Hi,

testing the Jetty MethodHandle usages we encountered this situation:

.class org.openjdk.mh.Main

Class klass = Main.class;
MethodHandles.Lookup lookup = MethodHandles.publicLookup().in(klass);
MethodHandle handle = lookup.findVirtual(klass, "test",
MethodType.methodType(String.class));


.module-info.java

module org.openjdk.mh {
 exports org.openjdk.mh to com.acme;
}


findVirtual() throws:
java.lang.IllegalAccessException: symbolic reference class is not
accessible: class org.openjdk.mh.Main, from
org.openjdk.mh.Main/noaccess (module org.openjdk.mh)

Removing the "to" clause from the exports in module-info.java fixes the issue.

Seems that a class cannot get a MethodHandle on itself, the reason
being that org.openjdk.mh is selectively exported in module-info.java.

Is this intended behavior?
I would have expected that a class could always get a MethodHandle on itself.
I tested this with 11, 15, and 16+25, all fail.

I have a simple reproducer, and I can open an OpenJDK bug.

It looks like you are starting with publicLookup and then teleporting to 
mh.Main that is not accessible to that lookup. When you change it from a 
qualified export  to exporting it to all modules then it becomes 
accessible to the lookup so that is why it works.


Have you tried starting with MethodHandles.lookup() ?

-Alan

Re: Why service provider method is called "provider", but not "provide"?

[Alan Bateman]

On 24/11/2020 06:46, Alex Orlov wrote:

Could you then explain the difference between service and service provider? As 
I understand
you’re saying they are the same.
The second paragraph of the class description attempts to establish the 
terminology. The "service" is the well known interface, "service 
providers" are the implementations, these are what ServiceLoader will 
locate and load. Also the "Designing services" an "Deploying service 
providers" sections are meant to provide guidance when designing for 
services.


-Alan

Re: Why service provider method is called "provider", but not "provide"?

[Alan Bateman]

On 23/11/2020 08:28, Alex Orlov wrote:
  
I don’t agree with that. It returns the object that is an instance of the service.
  

The method returns an instance of the service provider.

The enhancements to ServiceLoader in Java 9 were tracked as 
#ServiceLoaderEnhancements [1], the link to "Proposal" has a good 
summary of the update.


-Alan


[1] 
https://openjdk.java.net/projects/jigsaw/spec/issues/#ServiceLoaderEnhancements

Re: JPMS layer graph type

[Alan Bateman]

On 17/11/2020 19:44, Alex Orlov wrote:

Hello all,
  
I try to determine the type of JPMS layers graph and as I understand it is a directed acyclic graph.

Could anyone say if this is right or wrong.

Yes, the graph of module layers is a DAG.

-Alan.

Re: add-opens for dynamically created layers

[Alan Bateman]

On 16/11/2020 21:07, Alex Orlov wrote:

Hello Alan,
Thank you very much for your help. With ModuleLayer.Controller I 
solved the problem.
Do I understand it correctly — it is impossible to get reference to 
boot layer controller,

isn’t it?


Yes, that's right.

-Alan

Re: add-opens for dynamically created layers

[Alan Bateman]

On 16/11/2020 18:35, Alex Orlov wrote:

:
  
When I start my application I get:
  
java.lang.IllegalAccessError: class org.springframework.core.log.CompositeLog (in module spring.core) cannot access class org.apache.commons.logging.impl.NoOpLog (in module org.apache.commons.logging) because module org.apache.commons.logging does not export org.apache.commons.logging.impl to module spring.core.
  
To fix it, I wanted in the Framework dynamically addOpens to module (module.addOpens(...)), however, I got
  
Caused by: java.lang.IllegalCallerException: org.apache.commons.logging.impl is not open to module fw.module.
  
To fix it I added jvm argument:
  
--add-opens org.apache.commons.logging/org.apache.commons.logging.impl=fw.module \
  
However, it didn’t help. It seems to me that jvm arguments add-opens works only when the module that must open

its package is in boot layer, but in my situation fw.module that is in boot 
layer knows nothing about web application modules.
The static defineModules methods return a ModuleLayer.Controller 
that can be used to export or open any package of any module in the 
layer to any other module. It's the equivalent of using the 
--add-exports or --add-opens on the command line to export or open 
packages of modules in the boot layer. So in this case you should be 
able to use the ModuleLayer.Controller for (I assume) the "Web 
Application Layer" to export 
org.apache.commons.logging/org.apache.commons.logging.impl to 
spring.core. That can be a workaround using spring.core and 
org.apache.commons.logging agree sort out their differences on whether 
org.apache.commons.logging.impl is part of the supported API or not.


-Alan.

Re: How to do “--add-modules” with dynamically created layer in JPMS?

[Alan Bateman]

On 16/11/2020 09:17, Alex Orlov wrote:

Hello Alan,
Thank you for such detailed answer. I read it with attention, but 
still don’t know how to solve my situation.

Maybe may question didn’t provide all details, so here they are.
I want to create the following layer structure:
+++
+ Boot Layer +
+++
       |
       +
       + Web Server Layer +
       +
                         |
                         
                         + Web Application Layer +
                         
So, we have always one boot layer (it’s clear), one web server layer, 
and multiple web application layers. At
the same time all web application layers have one parent — web server 
layer. Boot layer doesn’t know and mustn’t know
about any child layer modules So it is impossible to add all 
modules to boot layer!
In web server layer I have servlet container (Jetty) and rest 
implementation (Jersey). When I create web server layer
Configuration cf = 
parentLayer.configuration().resolveAndBind(moduleFinder, 
ModuleFinder.of(), moduleNames);
ModuleLayer layer = parentLayer.defineModulesWithOneLoader(cf, 
parentClassLoader);
I add to it all modules of jetty, jersey and their dependencies. At 
the same time jetty knows nothing about jersey. When web server layer 
is created one of its modules starts Jetty.
So, when does jersey start? It starts only when web application layer 
is created with .war file that has web.xml with jersey servlet. For 
example:


    JerseyServlet
org.glassfish.jersey.servlet.ServletContainer

So, in such configuration, when web server layer is created no one can 
know if jersey will be used.
What what does JPMS do? It ignores all jersey modules including all 
its dependencies (including API). So, the question.

How to ask JPMS not to ignore modules that it must add to layer?


Thanks for the diagram, and names, of the module layers.

If I read your mail correctly then you are saying that the Jersey module 
is observable when you create the configuration for "Web Server Layer". 
It is not resolved because it's not in the set of root modules that you 
specify to resolveAndBind. It is also not required by any of the modules 
that end up in the configuration. This should not be a surprise. If the 
intention for "Web Server Layer" to contain Jersey and other modules 
that web applications may need then you will need to add them to the set 
of root modules ("moduleNames" in your code fragment. If the modules are 
completely unrelated then you might want to scan them so that 
"moduleNames" has the names of the modules that export an API.


Does the .war file containing "JerseyServlet" have a module-info.class? 
I'm curious if it declares "requires Jersey" as I would expect that the 
resolve or resolveAndBind for the web application will fail if it 
requires Jersey and it is not observable or is not in the parent layer.


-Alan

Re: How to do “--add-modules” with dynamically created layer in JPMS?

[Alan Bateman]

On 15/11/2020 19:16, Alex Orlov wrote:

Hi all,
  
I create JPMS layer this way:
  
Configuration cf = parentLayer.configuration().resolveAndBind(moduleFinder, ModuleFinder.of(), moduleNames);

ModuleLayer layer = parentLayer.defineModulesWithOneLoader(cf, 
parentClassLoader);
  
And I have a problem with adding implementation modules. These implementation modules are ignored by JPMS because classes from these modules are not used anywhere (it is clear, that classes from API modules are used). So, JPMS doesn't add these modules to layer.
  
As I understand, if I spoke about implementation modules on boot layer, I could use --add-modules jvm argument. However, I couldn't find any information how to force JPMS load my module (even if its classes are not used) for dynamically created layers.
  
Could anyone say how to do it?
It would be useful if you could give an example of "implementation 
module". I can't tell if you mean a jdk.* module, a service provider 
module that doesn't export an API, or something else.


In any case, I think the picture you are describing is a module that is 
observable, either in the run-time image or on module path. The module 
is not resolved at startup (and so is not in the boot layer) because the 
initial module that you specify with `java -m` doesn't transitively 
require it and it doesn't provide an implementation of a service that 
any of the other resolved modules use.


The summary is a module layer is immutable and there isn't any way to 
add modules to a module layer once it has been created. For what you are 
doing then maybe `--add-modules ALL-SYSTEM` can be used as workaround. 
This will ensure that all system/platform modules are in the boot layer 
as these modules cannot be loaded into a child layer (Johannes's suggest 
to specify the module name to resolveAndBind will not work for java.* 
modules as they cannot be loaded into child layers, his suggestion may 
work for other modules).


One other point to mention is that the original set of requirement 
envisaged some means to augment the set of platform modules at run-time. 
We didn't get to that requirement except for the limited scenario that 
is tools loading the JMX agent or a java agent into a running VM. These 
limited scenario involve loading the jdk.management.agent or 
java.instrument modules and that is some with a special child layer 
rather than augmenting the boot layer. This partial solution, along with 
`--add-modules ALL-SYSTEM`as a workaround for the general cases, has 
been sufficient to date.


-Alan

Re: @Generated requires java.compiler / what should my annotation processor do

[Alan Bateman]

On 02/11/2020 00:02, Rob Bygrave wrote:

*>  Since the type javax.annotation.processing.Generated has source *


* retention only, there are no @javax.annotation.processing.Generated
annotations in any class files compiled from source code emitted by your
processor*


The annotation processor generates *source code*. It does not generate byte
code. In this source generation case it does not seem to matter that the
annotation is retention source - it will not compile but result in
compilation errors [package javax.annotation.processing is not visible].

That is, with tooling of maven (and intellij) both do not compile the
generated source that includes javax.annotation.processing.Generated
without java.compiler being added to the *project module path*. It is not
sufficient that requires java.compiler; is in the module-info of the
annotation processor.

When the annotation processor generates java source code that includes
javax.annotation.processing.Generated and when this generated source is
compiled it fails to compile with compilation errors like:.


[ERROR] /.java:[4,24] package
javax.annotation.processing is not visible
[ERROR]   (package javax.annotation.processing is declared in module
java.compiler, but module example does not read it)


I believe the generated source is compiled along with the rest of the
project source code using the *projects module path* which should not
include the java.compiler module.

To me this compile error is correct and reasonable.  I think it's telling
me that the generated source should only include things that are in the
project module path (and java.compiler is not in the project module path).
If I read your mails correctly then there are three invocations of javac 
here.


The first is to compile the annotation processor that is module 
io.avaje.inject.generator. Its module declaration `requires 
java.compiler` and `provides javax.annotation.processing.Processor with 
...`, all good.


The second compilation is done with module io.avaje.inject.generator on 
the processor module path (--processor-module-path). This generates a 
source file with @javax.annotation.processing.Generated in the source 
file. It sounds like this is working.


I think the third compilation is compiling the generated source file. 
Your first mail suggests that it is not compiled into a named module (no 
module-info.java). This should work as all modules that export an API 
are resolved. However the compiler error that you show above looks like 
it is arises when compiling module "example". Is that a surprise? The 
error suggests that the module declaration for module example doesn't 
have `requires static java.compiler` so maybe the issue you are running 
into is in the POM or a tooling issue?


-Alan

Re: Debugging while using the module system

[Alan Bateman]

On 21/07/2020 19:48, Mantas Gridinas wrote:

Now that you mention it, this does occur while trying to use "Evaluate
expression" which spins up bytecode and evaluates it at run time. I
haven't contacted Jetbrains on this yet, but I'll open an issue on
their tracker as well.

This may be a good topic to continue on serviceability-dev as there may 
be additions to be JDWP and JDI to support some of these debugger 
use-cases. JVM TI has the module functions that are needed expand 
readability or the packages that are exported or opened for tools doing 
instrumentation and it's exactly the same support needed when generating 
code in the target VM to do expression evaluation. The summary is the 
it's not com.sun.jdi.VirtualMachine API or JDWP and it probably should.


-Alan.

Re: Debugging while using the module system

[Alan Bateman]

On 20/07/2020 21:25, Mantas Gridinas wrote:

Hi!

Recently I've moved my project to the module system from regular
classpath jars. Currently I'm using Adopt OpenJDK11 builds and for
debugging I provide the
-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=localhost:5005
command line argument while starting my application. As a debugging
client I use intellij idea's remote debugging functionality. Sometimes
(I am not sure myself what exactly causes this), while debugging my
project, I get into an issue where the debugger cannot access some
classes, their fields or methods. Since right now I cannot provide the
exact error message, I would like to ask: what are the limitations of
the debugger implementation, provided by the JDK, when used on a
runtime with a module system rather than a classpath based system?

Debuggers should work with modular applications. Under the covers, the 
JDWP agent that you specify with -agentlib uses the JVM Tool Interface 
and the Java Native Interface that aren't subject to access control. 
Maybe IDEA is spinning byte code or classes in the target VM? Have you 
tried submitting a bug to Jetbrains on this?


-Alan

Re: RFR JDK-8217527: jmod hash does not work if --hash-module does not include the target module

[Alan Bateman]

On 17/07/2020 19:45, Mandy Chung wrote:

Webrev:
http://cr.openjdk.java.net/~mchung/jdk16/webrevs/8217527/webrev.00/

The current implementation jmod hash ignores the target jmod file if 
specified.


Therefore even B requires A:

$ jmod hash --hash-module B --module-path jmods jmods/A.jmod

This jmod hash command finds no module to record hashes.

The patch fixes jmod hash such that if the target jmod file is 
specified, it is the module to record the hashes and it must have no 
outgoing read edges to the modules on the application path. If the 
target jmod file is not specified, jmod hash will find from the 
resolved modules that have no outgoing edges and match the given 
pattern to record hashes of the modules requiring it directly or 
indirectly.


jmod hash command should also report no hash recorded if no module 
found to record hashes.
This looks good good, and good to have it report when it doesn't record 
any hashes.


-Alan

Re: Force JPMS to add module to boot layer

[Alan Bateman]

On 22/06/2020 18:15, Alex Orlov wrote:

Hi Alan,
Thank you very much for your answer. In order to be concrete a test 
project was created

and it is here https://github.com/PashaTurok/hibernate-h2-test4
Besides, if it is not very difficult for you, could you answer 
separately to

each questions (# 1 and #2). So, I repeat them again:
1) Is it possible to force JPMS to add modules even if it think that 
they won’t be used

(by the way — nobody asks it).
2) As I understand this problem doesn’t appear when we create child 
layer and add any jars to it.

In this case JPMS doesn’t ignore any modules. Is this statement right?

java.lang.NoClassDefFoundError: 
net/bytebuddy/NamingStrategy$SuffixingRandom$BaseNameResolver
    at 
org.hibernate.cfg.Environment.buildBytecodeProvider(Environment.java:345) 
~[hibernate-core-5.4.15.Final.jar:?]
    at 
org.hibernate.cfg.Environment.buildBytecodeProvider(Environment.java:337) 
~[hibernate-core-5.4.15.Final.jar:?]
    at org.hibernate.cfg.Environment.(Environment.java:230) 
~[hibernate-core-5.4.15.Final.jar:?]


From what I can establish, org.hibernate.orm.core is an automatic 
module so it doesn't have any way to `requires net.bytebuddy` (an 
explicit module). You'll have to help out by adding `--add-modules 
net.bytebuddy` to the command line that surefire or failsafe uses.


I can't explain why you don't see issues when you use module layers. It 
is possible that you've specified "net.bytebuddy" in the list of root 
modules that you specify to the resolve method?


-Alan

Re: Force JPMS to add module to boot layer

[Alan Bateman]

On 21/06/2020 19:52, Alex Orlov wrote:

Hi all,
  
New version of maven-failsafe-plugin (3.0.0-M5) has been released and now it is possible to have two module-info in one

project — one in src/main/java and one in src/test/java  and to test modules on 
module -path.
Trying to use it I found the following.
  
Failsafe detects right module list using poms and adds it. The problem is that there are jars with module-info, with

Automatic-Module-Name and jars without anything. And if any module doesn’t show 
explicit dependency,
module is not added to boot layer by JPMS. For example, I have many hours story 
with hibernate and fasterxml.
In module path I saw:
/home/alex/.m2/repository/org/hibernate/hibernate-core/5.4.14.Final/hibernate-core-5.4.14.Final.jar
/home/alex/.m2/repository/com/fasterxml/classmate/1.5.1/classmate-1.5.1.jar
However, I got java.lang.NoClassDefFoundError: 
com/fasterxml/classmate/TypeResolver
Only later, I noticed that on boot layer I had only hibernate (without 
module-info) but
fasterxml (with module-info) was ignored by our lovely JPMS.
If hibernate is an automatic module, and if is indeed resolved, then 
every observable automatic module will also be resolved. Is there any 
way to get maven-failsafe-plugin to reveal the command line that it is 
using? Alternatively, is there any way to inject the 
"--show-module-resolution" option into the command line that it uses? 
That will help to discovered if hibernate is actually resolved.
  
So, it is necessary to add missed modules via test module-info to boot layer what is a very silly work:

1) you add to you pom dependencies that you don’t use
2) failsafe detects right dependencies using pom files and does it very well.
  
So, could anyone answer two my questions:

1) Is it possible to force JPMS to add modules even if it think that they won’t 
be used
(by the way — nobody asks it).
2) As I understand this problem doesn’t appear when we create child layer and 
add any jars to it.
In this case JPMS doesn’t ignore any modules. Is this statement right?
  
If you are add "--add-modules" to the command line then it will allow 
you to make sure that jackson is in the boot layer. That said, it may 
mask some other issue so finding a bit more on what 
maven-failsafe-plugin is doing will help the discussion here.


-Alan

Re: 8241770: Module xxxAnnotation() methods throw NCDFE if module-info.class found as resource in unnamed module

[Alan Bateman]

On 10/06/2020 22:22, Mandy Chung wrote:


Nit: A typo in the test s/reflectingly/reflectively/

102 * Test reflectingly reading annotations on a named module where 
the module
That typo was in two places and I only spotted one of them when creating 
the webrev. I'll fix the second one before pushing. Thanks!


-Alan

8241770: Module xxxAnnotation() methods throw NCDFE if module-info.class found as resource in unnamed module

[Alan Bateman]

This is a bug in Module.getXXXAnnotation implementation where it 
accidentally locates a module-info.class on the class path (say if 
someone deploys a modular JAR on the class path). An oversight in the 
original implementation. Simple fix to override loadClass to do the 
right thing. I've kept the changes as minimal as possible as this is a 
change that may need to be back-ported. The webrev with the change and 
updated test is here:


http://cr.openjdk.java.net/~alanb/8241770/webrev/

Thanks,
Alan

Re: Is it possible to add classes to JPMS module dynamically?

[Alan Bateman]

On 04/06/2020 13:34, Alex Orlov wrote:

Hi all,
  
Let's suppose we have a JPMS module  moduleA with the following  module-info :

module moduleA {
 exports modulea.generated;
}
Now, using javassist/byte-buddy we generate new  modulea.generated.Foo class. 
Could anyone say if we can add this class to  moduleA dynamically (at runtime) 
so as JPMS work with it as it works with module existing classes?
Yes, if you use Lookup.defineClass or some other means to define a class 
into one of moduleA's run-time packages then it be in moduleA.


-Alan.

Re: jlink 14.0.1 with 100 modules fail

[Alan Bateman]

On 30/05/2020 17:40, Christian Stein wrote:

Hi,

the modular [bach-demo-99] I generated to check my build tool
against, failed to create a custom runtime image using jlink
via itsToolProvider-based service entry-point.

The project consists of 100 module descriptors. No other Java
source files (for classes and interfaces) are part of that demo.
Each descriptor ball${i}n declares a "requires ball${j}n;" with j
set to a number from 0 to i-1 ... in words, all lower modules.

javac compiles this project with problems.

Arguments passed to jlink (as an array of string) reads:

jlink
   --add-modules ball27n,ball43n,ball88n,[...],ball77n,ball93n
   --module-path .bach\workspace\modules
   --output .bach\workspace\image
   --compress 2
   --no-header-files
   --no-man-pages

The error message reads:

jdk.internal.org.objectweb.asm.MethodTooLargeException: Method too large:
jdk/internal/module/SystemModules$all.moduleDescriptors
()[Ljava/lang/module/ModuleDescriptor;

Is there a limitation of supported modules that can be linked?
Is the module graph of this conceived project too far out of scope?

Shall I open an issue at JBS?

Yeah, this is an issue with the jlink system modules plugin where it 
needs to split the initialization when it results in a method size more 
than 64k. It it tracked by JDK-8240567. Can you add `--disable-plugin 
system-modules` to the jlink command as workaround until this is fixed?


-Alan

Re: RFR 8246034: Remove java.base/share/classes/jdk/internal/jrtfs/jrtfsviewer.js and java.base/share/classes/jdk/internal/jrtfs/jrtls.js

[Alan Bateman]

On 28/05/2020 07:14, sundararajan.athijegannat...@oracle.com wrote:

Please review.

Bug: https://bugs.openjdk.java.net/browse/JDK-8246034

Webrev: http://cr.openjdk.java.net/~sundar/8246034/webrev.00/
Have you checked if there are any make file configured to copy .js 
files? Otherwise looks good.


-Alan

Re: JPMS module-path and layers

[Alan Bateman]

On 19/05/2020 09:00, Alex Orlov wrote:

Hi all,
  
I want to understand all details of module-path and layers in java9+ and I have two questions:

*  As I understand modules on module-path are added automatically only to boot 
layer and not added automatically to any other layers. I mean, if any child 
layers are created module-path will be ignored for them. Is this correct?
*  Not all modules from module-path are added to boot layer, but only main 
module and dependent modules. For example, if we have  A ,  B ,  C ,  D modules 
on module-path and  A is main module and uses  B ,  B uses  C , then  D module 
won't be added to boot layer. Is this correct?
Could anyone help me with these questions?

Have you been through the java.lang.module package description and the 
APIs for Configuration.resolve? I think the main thing to understand is 
that A, B, C, and D are observable when doing the module resolution and 
creating the readability graph for the boot layer. D is not in the 
readability graph because it's not required by any of the other modules 
in the graph. Later you will use Configuration.resolve to do module 
resolution for what will become the child layer. The module finders that 
you specify to the resolve method will determine if D is observable or 
not. Depending on the root modules that you specify to resolve, then it 
might be that D is the readability graph for the child layer.


-Alan

Re: JavaCompiler called from JPMS module

[Alan Bateman]

On 16/05/2020 17:53, Alex Sviridov wrote:

:
  
  
When H2 is on classpath I don't have any problems - trigger is compiled. However, when H2 is on

module path (H2 has in manifest Automatic-Module-Name: com.h2database) on some 
of child layers
(not boot layer) I get the following `output`:
  
     /org/h2/dynamic/trigger/SOME_TRIGGER.java:6: error: package org.h2.api does not exist

     public static org.h2.api.Trigger create() {
     ^
     1 error
  
As I understand the problem is with classloader and JavaCompiler. Could anyone say how it can be fixed?


You can't compile against modules in module layers. Instead you'll need 
to specify the module path where the compiler can find the JAR file with 
module com.h2database. This snippet might help:


    Path src = ...   // path to the source code for SOME_TRIGGER
    Path dir = ...   // directory that contains com.h2database
    Path output = ... // output directory

    JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
    StandardJavaFileManager fm = 
compiler.getStandardFileManager(null, null, null);
    fm.setLocationFromPaths(StandardLocation.MODULE_PATH, 
List.of(dir));
    fm.setLocationFromPaths(StandardLocation.CLASS_OUTPUT, 
List.of(output));
    Iterable paths = 
fm.getJavaFileObjects(src);
    JavaCompiler.CompilationTask task = compiler.getTask(null, fm, 
null, null, null, paths);

    task.addModules(List.of("com.h2database"));
    task.call();

The call to addModules is needed because SOME_TRIGGER isn't being 
compiled as a module so there is nobody to require com.h2database.


-Alan

Re: Review Request 8240910: jmod rejects duplicate entries in --class-path jars

[Alan Bateman]

On 07/05/2020 19:39, Mandy Chung wrote:
I have taken a further step to clean this up by recording the written 
entries such that it can filter out duplicated entries properly:


http://cr.openjdk.java.net/~mchung/jdk15/webrevs/8240910/webrev.01/
This version looks okay to me as it cleans up the original code for the 
exploded case too.


-Alan

Re: RFR: JDK-8241602 jlink does not produce reproducible jimage filesJ

[Alan Bateman]

On 06/05/2020 15:45, Jim Laskey wrote:

http://cr.openjdk.java.net/~jlaskey/8241602/webrev-02

This version looks okay to me.

-Alan.

Re: RFR: JDK-8241602 jlink does not produce reproducible jimage filesJ

[Alan Bateman]

On 06/05/2020 14:42, Jim Laskey wrote:

:

Aside: The order in the file is still somewhat scattered, based on 
archive and archive content.  We have the a pattern based sorting 
plugin which we don't use and we never did any locality vs performance 
testing.


Not in the test but the JDK does use the order-resources plugin with the 
following pattern:


JLINK_ORDER_RESOURCES += \
    /java.base/java/** \
    /java.base/jdk/** \
    /java.base/sun/** \
    /java.base/com/** \
    /jdk.localedata/** \
    #

Anyway, I looked at webrev-01 and the jlink changes look good. The 
updated test doesn't use the jimage API so no need to open 
jdk.internal.jimage. The imports can be removed too, also RUNTIME is no 
longer used in this version. Have you decided on a test name? The 
rallying call is "reproducible builds" and I'd prefer to have 
"Reproducible" in the test name so its consistent with the existing test 
that checks for reproducible with user modules.


-Alan.

Re: RFR: JDK-8241602 jlink does not produce reproducible jimage files

[Alan Bateman]

On 05/05/2020 20:56, Jim Laskey wrote:

This fix addresses the inconsistent ordering by jimage content by jlink from 
run to run. Bottom line, the implementer was using HashSet without defining 
hashcode/equals for the Set entry classes.

webrev: http://cr.openjdk.java.net/~jlaskey/8241602/webrev-00 

jbs: https://bugs.openjdk.java.net/browse/JDK-8241602 


DirArchive and JmodArchive look okay. They could use Objects.hash/equals 
but what you have is okay too. Can you re-check JarArchive as it is 
created with a runtime version so equals should be checking 3 fields.


The existing test for reproducible builds is JLinkReproducibleTest. 
Adding a new test is okay too but we should at least try to keep the 
names consistent. A couple of suggestions for the test in the webrev:
- the @modules tag needs to include java.desktop as it is needed by the 
test. Better still would be to change to java.se so that there are more 
modules in the images.
- Did you mean to open jdk.tools.jlink.internal? Maybe its a leftover 
from a previous iteration of the test?
- You can use Files.mismatch to compare the lib/modules files are 
identical (like JLinkReproducibleTest).  It's okay to use the jimage 
ImageReaderFactory to check the names tables too but I think it's more 
important to check that the lib/modules files are identical before 
probing further.


-Alan

8243596: ModuleLayer::parents should return an unmodifiable list

[Alan Bateman]

This is an small oversight and inconsistency that crept in when the 
ModuleLayer was updated to support multiple parents. The list of parent 
layers returned by the parents() method should be specified (and 
implemented) to return an unmodifiable list. It currently returns a 
modifiable list. The equivalent Configuration::parents is correctly 
specified, as is the Configuration::modules and ModuleLayer::modules 
methods.


The change proposed here fixes the javadoc and implementation. The 
compatibility impact is minimal as nothing could depend on being able to 
dynamically change the list of parents (as it can't be changed dynamically).


I've extended the existing tests to ensure that ::modules, 
Configuration::parents, ModuleLayer::modules and ModuleLayer::parents 
return unmodifiable collections. In passing, there is also a typo in the 
Configuration::modules where the javadoc says "immutable" instead of 
"unmodifiable".


http://cr.openjdk.java.net/~alanb/8243596/webrev/

The patch doesn't update the date in the copyright headers, I'll update 
those before pushing after the changes (and CSR) are agreed.


-Alan

8243666: ModuleHashes attribute generated for JMOD and JAR files

[Alan Bateman]

This is build reproducibility issue with the module-info.class files in 
the packaged modules (JMOD files in the case of the JDK build).


One part to this is the ModulePackages class file attribute that is 
added by the tools, including the jmod tool and the AddPackagesAttribute 
tool in the JDK build. The package names are added in iteration order so 
can vary from run to run. This is trivially fixed to sort the package 
names.!


The other part is the ModuleHashes attribute where a hash is computed 
from all contents of a package module. In the case of JMOD files this 
means the hash covers the timestamps on the entries in the JMOD so it 
varies from build to build too. The proposal is to compute the hash over 
the names and content of the entries. The timestamps and other 
non-interesting meta data is not included in the hash. Hashes are to 
avoid accidental mixing of tightly coupled modules from different builds 
so I think this should be strong enough.


The webrev with the proposed changes is here:
   http://cr.openjdk.java.net/~alanb/8243666/webrev/

There are follow-on issues that will need to be fixed in jlink and the 
image writer before the modules container is reproducible. That is 
something for another set of patches.


-Alan

Re: RFR (T): 8242846: removed an empty file test/jdk/tools/jlink/plugins/OrderResourcesPluginTest.java

[Alan Bateman]

On 21/04/2020 04:58, Ao Qi wrote:


On 2020/4/20 下午9:27, Alan Bateman wrote:

On 20/04/2020 11:32, sundararajan.athijegannat...@oracle.com wrote:

Hi Alan,

I don't remember it now. Likely a mistake. The changeset

http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/a066fe7b1b42

has that file. Perhaps it may be useful to restore & check if the 
test passes.
Yes. I think it was the only test for this jlink option so let's 
bring it back or replace it.




I think the test should be brought back. I restored the test and it 
didn't pass. Here is the diff against the original test[1]: 
http://cr.openjdk.java.net/~aoqi/8242846/diff.01, and the new webrev 
is http://cr.openjdk.java.net/~aoqi/8242846/webrev.01/ . I'm not 
familiar with this field, so I am not sure I fix this rightly.

Thanks, I think this looks okay.

-Alan.

Re: BackEnd Service Provider.

[Alan Bateman]

On 21/04/2020 14:01, Doug Simon wrote:

Hi Gary,

I cannot understand why --add-reads does not work and have reached out 
to Alan Bateman for more insight.



I read through the thread [1] and I don't see an obvious solution to this.

The main issue here is that org.grfstuff.compiler wants to provide an 
implementation of org.graalvm.compiler.hotspot.HotSpotBackendFactory but 
that service type is not in org.grfstuff.compiler and 
org.graalvm.compiler.hotspot is not exported to org.grfstuff.compiler by 
any of the modules that it reads. The post resolution consistency check 
that it trips up on is specified in the Configuration API docs if you 
are interested.


I assume `--add-exports 
jdk.internal.vm.compiler/org.graalvm.compiler.hotspot=org.grfstuff.compiler` 
was added in an attempt to workaround this but that CLI option is the 
equivalent of jdk.internal.vm.compiler invoking Module::addExports after 
the module graph has been reified. It doesn't impact resolution or any 
of the post resolution consistency checks at run-time. Yes, there is an 
unfortunate difference between between compile-time and run-time in this 
regard but that is something for another discussion.


I don't think there are any obvious workarounds. The Module API defines 
addXXX methods to export or open packages at run-time, extend 
readability or add a service dependences but it doesn't define 
addProvides. I think it came up once during JDK 9 but more in the 
context of symmetry rather than a compelling use-case. It's not clear 
that the issue under discussion here is compelling enough, needs more 
thought as the scenario is a bit weird. If org.grfstuff.compiler was co 
developed with jdk.internal.vm.compiler then it could use a qualified 
export of course.


Initially I thought you could workaround this by moving 
org.grfstuff.compiler  to the class path but this it's not going to work 
as ServiceLodaer usage in Graal seems to use module layers as congtext, 
so it will not find implementations on the class path.


-Alan

[1] https://mail.openjdk.java.net/pipermail/graal-dev/2020-April/005949.html

Re: RFR (T): 8243117: Cleanups in Java code of module jdk.jlink

[Alan Bateman]

On 19/04/2020 20:08, Langer, Christoph wrote:

Hi Alan,


Looks okay to me too, except @SuppressWarnings("unused") looks strange,
is that an Eclipse compiler warning name?

Yes, it comes from Eclipse, probably only used by its compiler. The private 
field EXIT_SYSERR doesn't seem to be used. Alternatively, I could comment it 
out? Or remove it?

The jimage tool went through a few iterations and I think it's just a 
left over and should be okay to remove.


-Alan

Re: RFR (T): 8242846: removed an empty file test/jdk/tools/jlink/plugins/OrderResourcesPluginTest.java

[Alan Bateman]

On 20/04/2020 11:32, sundararajan.athijegannat...@oracle.com wrote:

Hi Alan,

I don't remember it now. Likely a mistake. The changeset

http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/a066fe7b1b42

has that file. Perhaps it may be useful to restore & check if the test 
passes.
Yes. I think it was the only test for this jlink option so let's bring 
it back or replace it.


-Alan.

Re: RFR (T): 8243117: Cleanups in Java code of module jdk.jlink

[Alan Bateman]

On 19/04/2020 17:31, Claes Redestad wrote:

Looks good to me, Christoph,

JmodTask.java: Not sure what the consensus is about wildcard imports,
but I'm fine with it here.
Looks okay to me too, except @SuppressWarnings("unused") looks strange, 
is that an Eclipse compiler warning name?


-Alan

Re: Cross module single ServiceProvider for JPMS services

[Alan Bateman]

On 19/04/2020 16:05, Uwe Schindler wrote:

Hi Alan,

from what I understood: He has some utility method in a shared class that wraps 
ServiceLoader to discover services and do something with it (the usual way to 
prevent code duplication). Problem is that ServiceLoade.load() is 
caller-sensitive, so a shared utility method only sees the services that the 
modules is allowed to see.

I answered on this in another mail: He can have a shared utility class, but the 
ServiceLoader instance should NOT be instantiated inside the utility class. 
What you suggest is to additionally open the shared module to allow reading 
services, which looks wrong.

Unfortunately, the caller-sensitiveness of ServiceLoader.load() cannot easily 
be seen from the Javadocs. IMHO there should be some improvements on this. I 
was stumbling on this, too.

The ServiceConfigurationError specified by each of the load methods does 
try to make it clear it will be thrown when the module does not declare 
that is uses the service. The exception should be clear too. I agree 
that addUses is not easy to find but it is right method for these scenarios:


"This method is intended for use by frameworks that invoke ServiceLoader 
on behalf of other modules or where the framework is passed a reference 
to the service type by other code. This method is a no-op when invoked 
on an unnamed module or an automatic module."


So it's just adding a use-edge at run-time, it's not opening any 
packages or changing the readability.


-Alan.

Re: Cross module single ServiceProvider for JPMS services

[Alan Bateman]

On 19/04/2020 15:00, Alex Sviridov wrote:

Hi all,
  
Let's suppose we have modules (A, B, C..) and every module consumes and provides services.

To find these services I want to create one util class, for example 
ServiceProvider that will be used
by all modules to get necessary services. There are two ideas:
*  to have one point to control providing services within system
*  to avoid code duplication.
However, I can't do it. The problem is that if we put ServiceProvider, for 
example, in module A,
then it can find only services that module A uses (service is declared in  uses 
in module A module-info).
Is it possible to do, if yes, then how?
If I read this correctly then we have code in module A looking to load 
service provider of type S but A's module declaration doesn't declares 
`uses S`. Have you tried the Module addUses method? That will add the 
"use" edge at run-time that should make this work.


-Alan.

Re: Loading an automatic module from an exploded directory

[Alan Bateman]

On 18/04/2020 23:04, Eirik Bjørsnøs wrote:

:

Could we update ModulePath could to detect "automatic module 
directories"?. Any directory path name ending with ".jar" could be 
loaded as an automatic directory module, with the same semantics as if 
the directory was zipped into a JAR.


This would require a refactoring of deriveModuleDescriptor to allow it 
being called both from readJar and readExplodedModule. 
Then readExplodedModule would need an update to detect the lack of 
module-info.class, and call deriveModuleDescriptor just like readJar 
does. scan(Path) would need an update to make it skip directory 
scanning on directories ending with ".jar".


As far as I can tell, this would be (or allow) an implementation of 
MultiModuleJARs.


Directories with a ".jar" suffix is clever but I don't think it gets 
away from the substantive issue that Java SE only specifies how 
automatic modules come into being when they are packaged as JAR files. I 
could also imagine tools and libraries that scan tripping up on 
directories named ".jar". There are also many libraries that peek into 
the JAR manifest with the java.util.jar APIs so they will be surprised too.


Just on MultiModuleJARs (and its buddy #MultiModuleExecutableJARs). This 
would probably be significant enough to need a JEP. It would need to 
explore whether the modules in the JAR are observable when the JAR file 
is on the module path, this amounts to deciding if they are resolved and 
loaded into the boot layer or child later. If observable on the module  
path then it mean javac would need to support it. Automatic modules 
would bring many discussion points to the table. One of their important 
roles of automatic modules is to bridge to the class path so what does 
not mean when everything is packaged into the same JAR file, does the 
JAR file have a directory with the "class path" too? There are 
implications for jlink and several other tools (jar) aswell. So in 
summary, and at least in my view, there is a lot more exploration needed 
before doing just implementation changes to support "exploded automatic 
modules" in JAR files.


-Alan

Re: Loading an automatic module from an exploded directory

[Alan Bateman]

On 10/04/2020 17:48, Eirik Bjørsnøs wrote:


I once had the pleasure of debugging a system which after running in 
production for a few days would suddenly start returning 404 errors.


Turned out someone (me) had forgotten to specify where Jetty should 
unpack its WAR files. So they ended up on shared temp space where a 
weekly Centos cron job cleaned them up.


After this experience I pledged to never unpack code to temp again :-)

Also, the performance implications seems not very appealing.
If MultiModuleJARs [1] is re-visited then it would need to explore this 
topic a bit more. For now, the only case where the JDK implementation 
unpacks is when ModuleFinder.of is created with a file path to JAR file 
in a custom file system or to a directory in a custom file system that 
contains JAR files. These JAR files might be signed so this is why 
JarFile (with the JAR verifier support) is used. It would be possible to 
do the verification without support for the JarFile infrastructure, it 
just wasn't a priority in JDK 9.


-Alan

[1] https://openjdk.java.net/projects/jigsaw/spec/issues/#MultiModuleJARs

Re: RFR (T): 8242846: removed an empty file test/jdk/tools/jlink/plugins/OrderResourcesPluginTest.java

[Alan Bateman]

Sundar - do you recognize this? I assume truncating this test to 0 bytes 
was a mistake but I can't tell if the test should be brought back or not 
(to ensure that`jlink --order-resources` is tested).


-Alan

On 17/04/2020 11:34, Ao Qi wrote:

Hi,

The original email is waiting for moderator approval. Subscribed to
jigsaw-dev and resent the RFR (cc'ed core-libs-dev).

Thanks,
Ao Qi

On Wed, Apr 15, 2020 at 10:11 PM Ao Qi  wrote:

Hi all,

test/jdk/tools/jlink/plugins/OrderResourcesPluginTest.java was changed
to an empty file by JDK-8162538, but not removed. I think we should
remve this file.

JBS: https://bugs.openjdk.java.net/browse/JDK-8242846
webrev: http://cr.openjdk.java.net/~aoqi/8242846/webrev.00/

Thanks,
Ao Qi

Re: Classloading with multiple parent layers JDK11 vs JDK14

[Alan Bateman]

On 17/04/2020 21:11, Alex Sviridov wrote:

Hi all,
  
I have the following layers:
  
LayerA (moduleA)                   LayerB (moduleB)

              |_|
                                  |
                    LayerC(moduleC)
  
LayerC has two parent layers layerA and LayerB and moduleC requires moduleA

and moduleB. When I start my application with JDK 11/14 I get no resolving 
errors
and everything is ok. However, when I use JDK 11 in moduleC when I use
application I get  java.lang.NoClassDefFoundError moduleB.Foo.class,
when I run with JDK 14 I don’t get such exception. The JDK I use listed
below.

Is moduleB an automatic module? If so then you are probably running into 
JDK-8211825 [1].


-Alan.

[1] https://bugs.openjdk.java.net/browse/JDK-8211825

Re: RFR 8242860: test/jdk/tools/jlink/ModuleNamesOrderTest.java uses nashorn module

[Alan Bateman]

On 17/04/2020 07:21, sundararajan.athijegannat...@oracle.com wrote:

Hi,

nashorn modules were used only as an example in this test. Using 
jdk.jshell module instead to test

known module dependencies in jlink produced images.

Please review.

Bug: https://bugs.openjdk.java.net/browse/JDK-8242860
Webrev: http://cr.openjdk.java.net/~sundar/8242860/webrev.00/

Looks okay.

Re: RFR: 8242452: During module definition, move conversion of packages from native to VM

[Alan Bateman]

On 16/04/2020 00:04, Ioi Lam wrote:

Hi Claes,

This is very good optimization. I have a few nit picks:

[1] I think the following is unnecessary. The original 
Java_java_lang_Module_defineModule0 and GetInternalPackageName 
function did not perform null check or type check. I think this should 
be changed to an assert.
It would be a bug if these JVM_XXX functions were called with null or 
otherwise invalid values. So additional checks shouldn't be needed, they 
would be just a safety net. That said, I thought the white box tests 
covered these cases so that the JVM functions could be tested with junk 
input to make sure that they were robust.


-Alan

Re: How does classloading work with multiple child layers?

[Alan Bateman]

On 15/04/2020 12:45, Alex Sviridov wrote:

Hi Alan,
  
Thank you very much for such detailed answer! Only one note — is the following

paragraph right?
   «When Child2 is created it will create L2 and map moduleC to L2. When
code in moduleC tried to resolve a reference to a class in its own
module then it will be loaded by L1 (POINT X) (no delegation). When moduleC
references a b.* class then it will be delegate to L2 (POINT Y) to resolve the
reference. When moduleC references a class in java.base then L1 will
delegate to the boot loader.»
  
Maybe  POINT X = L2 and  POINT Y = L1 ???



Sorry about, I don't know how they got transposed.

In any case, I hope it is clearer how these class loaders delegate.

-Alan

Re: How does classloading work with multiple child layers?

[Alan Bateman]

On 14/04/2020 20:48, Alex Sviridov wrote:

Hello everybody,
  
I try to understand how classloading works when there are multiple child layers

and have a problem.
  
Let's consider the following situation. There are three layers.

BootLayer (moduleA)
  |
  |___Child1(moduleB)
     |
     |__ Child2(moduleC)
  
Child1 is the child of the BootLayer, Child2 is the child of the Child1.
  
Child1 and Child2 were created with the same code:

ClassLoader parentClassLoader = ClassLoader.getSystemClassLoader();
ModuleLayer layer = parentLayer.defineModulesWithOneLoader(cf, 
parentClassLoader);
  
As you see the parent class loader of both child layers is SystemClassLoader.
  
I am a beginner at classloading, but I've read about parent-first delegation model. However, if for both child layers SystemClassLoader is used, then why they see classes from other layers? I mean, why does moduleC

see classes from moduleB if the parentClass is SystemClassLoader 
(SystemClassLoader as I understand
delegates to BootLayer, but not to Child1 layer). Could anyone explain?

Module layers are an advanced topic. ClassLoaders are also an advanced 
topic. When working with module layers and using the 
defineModulesWithXXX methods to create the module layers, then you 
mostly don't need to be too concerned with class loaders. They are still 
used to load classes but they are mostly in the background (and not in 
your face).


You also don't need to be too concerned with the "parent class loader" 
that you specify to the defineOneWithOneLoader method. It's not used 
when loading classes from modules, it's only when for cases where the 
code in moduleB or moduleC tries to load a class that is not in a 
module, maybe Class.forName("Foo") where Foo is on the class path. So 
probably best to ignore the parent class loader when starting out.


The API docs [1] explain how the delegation works with modules but maybe 
it's not clear enough for what is needed here. In your example, support 
L1 is the class loader for module B in child layer 1, and L2 is the 
class loader for module C in child layer 2. Further suppose the module 
declarations are:


module moduleC {
    requires moduleB;
}

module moduleB {
    exports b;
}

The configuration for Child1 is very simple: one moduleB that reads 
java.base


The configuration for Child2 is also very simple: one moduleC that reads 
moduleB and java.base.


When Child1 is created it will create L1 and map moduleB to L1. When 
code in moduleB tries to resolve a reference to a class in its own 
module then it will be loaded by L1 (no delegation). When moduleB 
references a class in java.base then L1 will delegate to the boot loader.


When Child2 is created it will create L2 and map moduleC to L2. When 
code in moduleC tried to resolve a reference to a class in its own 
module then it will be loaded by L1 (no delegation). When moduleC 
references a b.* class then it will be delegate to L2 to resolve the 
reference. When moduleC references a class in java.base then L1 will 
delegate to the boot loader.


If you draw this out then you should see that the class loader 
delegation is "direct delegation" and exactly mirrors the edges in the 
readability graph (Configuration object).


Hopefully this is enough to get your started. It really needs diagrams 
and graphs to explain some of these details probably. As I said, you can 
mostly ignore class loader details when working with module layers.


-Alan

[1] 
https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/lang/ModuleLayer.html#defineModulesWithOneLoader(java.lang.module.Configuration,java.util.List,java.lang.ClassLoader)

Re: excluding transitive module

[Alan Bateman]

On 14/04/2020 09:24, Jochen Theodorou wrote:

Hi all,

I am wondering if there is a solution purely in the module-info for this:

* Project requires Library1 and Library2
* SomeLibrary requires SharedApi
* OtherLibrary requires SharedApiImpl

The problem is, that it will not compile because SharedApi and
SharedApiImpl implement the same packages but not the same logic. One is
just an API, while the other is an implementation of the API.

This seems futile without first cleaning up the architectural issues.

How does SharedApi locate the implementation? Is this configurable or 
does it always assume it's in the same run-time package as itself? If 
the API and implementation (or default implementation) have to be in the 
same run-time package then it leads to SharedApi and SharedApiImpl 
needing to be combined into one module, not two.


Why is OtherLibrary requiring SharedApiImpl? This suggests that 
SharedApiImpl is more than an implementation, does it have an additional 
implementation specific API that OtherLibrary make use of?


If you decide to combine the API and default implementation into the one 
module then it will look something like this:


module api {
    exports api;
    uses spi;
}

The default implementation will be non-public classes in the same 
package as the public API classes. The `uses spi` is stand in for 
whatever the service type that some other implementation can provide 
(assume it is indeed pluggable, why else would they be separated in the 
first place?).


If you don't want to combine the API and implementation into the same 
module then you'll have to move the implementation to another package so 
you have something like the following fully encapsulated implementation:


module impl {
    requires api;
    provides spi with impl;
}

Different run-time packages means they can't use package-privates. If 
there is an implementation-specific/extension API then impl will need to 
export that package. There will be presumably be api types in the impl 
specific API so the requires would change to `requires transitive api`. 
OtherImpl will `requires impl`.


-Alan

Re: Loading an automatic module from an exploded directory

[Alan Bateman]

On 10/04/2020 11:14, Eirik Bjørsnøs wrote:

:
Since ModuleReader already has the list() method, code in java.lang.module
could use that to scan for packages and service names it needs to derive
automatic ModuleDescriptors.

Clients would now only need to pass the custom module reader and a default
automatic name:

ModuleFinder moduleFinder = ModuleFinder.of(ModuleReader moduleReader,
String defaultAutomaticModuleName);

I would still have to create my custom module reader, but I would no longer
need to know or care about the details of how automatic modules work.

Something to consider?

This would require the API specify how automatic modules are created 
from the contents of "something" that already have a ModuleReader. 
Specification could be crafted of course but I have concerns that it 
goes beyond the original intention of automatic modules. That is, the 
original intention was to allow yesterday's JAR files be used as today 
modules. More sophisticated applications have the APIs to create 
automatic modules for other cases where needed. So I think it needs a 
bit more thought on whether additional APIs should be exposed.


-Alan.

Re: Loading an automatic module from an exploded directory

[Alan Bateman]

On 10/04/2020 10:53, Eirik Bjørsnøs wrote:

:

Is it safe to assume that all access to content with a module happens 
through the ModuleReader?


Yes, the ModuleReader is the only way to access the contents of a module 
so you'll see all class loading, finding resources, etc. all go through 
the reader. We can't stop nosey code opening the packaged modules 
directly of course but none of the standard APIs bypass the module reader.


-Alan

Re: Loading an automatic module from an exploded directory

[Alan Bateman]

On 09/04/2020 19:23, Eirik Bjørsnøs wrote:


Alan,

If I read your mail correctly, you are creating "multi-module JAR
files"
where the modules are "exploded" under /META-INF/modules in
${NAME}-${VERSION} directories.


Correct.
Do they need to be "exploded"? If the dependences are automatic modules 
then I assume putting the JAR files (without unpacking) into 
META-INF/modules should just work without needing a new ModuleFinder 
implementation or implementing the exploded equivalent of multi-module 
JAR files (or modular multi-release JAR files).


(The reason it should work is that the built-in implementation can work 
with paths to locations in virtual file systems. It handles this by 
copying the JAR files from the virtual file system to the default file 
system so that it can be opened with the JarFile API. It does this in 
order to support signed JARs as the zip file system provider doesn't 
have any support for validating signatures. It's possible this approach 
isn't very efficient, also we haven't looked into interactions with the 
cron jobs that clean tmpfs so there may be a bit more work needed here).




To achieve this, I had to copy / replicate quite a bit of code from 
the ModulePath class which is not accessible outside java.base.


It wasn't all that hard, I just wasn't happy with the amount of 
classes / methods I had to copy from java.base only to slightly modify 
them


This includes:

Code to create / build the ModuleDescriptor (stolen from 
deriveModuleDescriptor)
Code to clean the artifactId used to produce the automatic name 
(Stolen from cleanModuleName, Patterns and Checks)

Duplication of ExplodedModuleReader which also dragged in Resources.
Understood although you probably don't need a deep copy of everything as 
there is a lot in exploded module reader related to potential security 
and traversal on the default file system that is less interesting in JAR 
files.




:

The runtime loads a graph of module layers, each containing one or 
more modules. Layers may depend on other layers, forming a DAG with 
corresponding class loader delegation.


When the runtime detects that a module needs to be redeployed, the 
transitive closure of depending layers is removed from the graph and 
new module layers are created and added to the graph.


Services are bound using dependency injection, so there's also a DAG 
of service dependencies. This allows restarting services which are 
affected by the redeploy, either because they are provided from 
updated modules or because they consume services from updated modules.


So not really hot-deploying in the JVM sense, but still pretty fast 
and developer-friendly.


My demo has a module which provides a JAX-RS resource which is 
consumed by a Jersey module which provides an HttpServlet which is 
again consumed by a Jetty module which deploys the servlet in a 
servlet context.


Redeploying the module containing the JAX-RS resource takes something 
like 50ms IIRC.


This sounds very interesting. I'm sure there are several people here 
that would be interesting in seeing a write-up or demo of this. I'm 
curious if the service wiring make use of the existing uses/provides or 
something else. I'm also curious if you've run into any issues with 
multi-parent configurations as that is an area that might need attention 
some day.


-Alan

Re: Loading an automatic module from an exploded directory

[Alan Bateman]

On 09/04/2020 16:42, Eirik Bjørsnøs wrote:

The current implementation of automatic modules seems to assume that an
automatic module is always packaged as a jar file.

I'm working on a module runtime where this is not always the case, and the
limitation has become a bit of a challenge.

I want to package applications (modules + runtime) at build time into a
single jar for distribution.

The runtime loads modules from directories within its own jar.  This means
that ModuleFinder.of(Path..) receives module locations in the form:

jar:file:///path/to/single.jar!/META-INF/modules/module-1.0/

This works fine as long as modules are explicit. (With the unrelated
limitation that the multi-release feature also seem to assume jar files)
The only packaging format for automatic modules that Java SE defines is 
JAR files. The "Multi-release JAR files" feature is also JAR file only.


If I read your mail correctly, you are creating "multi-module JAR files" 
where the modules are "exploded" under /META-INF/modules in 
${NAME}-${VERSION} directories. It shouldn't be too hard to create your 
own ModuleFinder that finds modules under META-INF/modules. This would 
mean implementing ModuleFinder rather trying to use 
ModuleFinder.of(Path...). I assume you've found ModuleDescriptor.read to 
read/parse the module-info.class of explicit modules. You are right that 
it would require code to scan directory trees, at least the equivalent 
of automatic modules, maybe for explicit modules too. However, it 
shouldn't be too hard. Have you tried the zip file system provider? That 
would allow you to open the JAR file as a file system so you can use the 
file system API.



:

I have also identify an additional use case which is to allow hot-deploying
automatic modules during development from target/classes using a Maven
plugin.

I'm not sure how to interpret this but just to say that the unit of 
replacement is the module layer, you can't replace modules in a layer 
and/or dynamically change the set of packages in a loaded module.


-Alan

Re: missing resources when using --patch-module

[Alan Bateman]

On 03/04/2020 12:46, Tom De Wolf wrote:

Hi,

I understand that for individual classes and non-java resources a 
patch should replace them. However for directories which are actually 
java packages this seems to be odd. Does this not make it impossible 
to add a new class to an existing package or replace only 1 class in a 
package but keep the rest? If only the package found in the patch is 
returned then only classes in that patch are found and not the classes 
in the original package.
Best to read the "Patching module content" section of JEP 261 [1] as it 
has the details. Hopefully it will be clear that you are add classes or 
resources to an existing package (the classes loaded from that location 
will be in the same run-time package as those located from the 
equivalent location in the original module).


Just on directories as resources. This is a very under-specific area and 
has always been implementation specific as to whether directories are 
located as resources, also whether they are synthesized when they don't 
exist (e.g. a JAR file might have directory entries stripped, zip -D, 
etc). Short comings in the specification of resources pre-date modules.


-Alan

[1] https://openjdk.java.net/jeps/261#Patching-module-content

Re: missing resources when using --patch-module

[Alan Bateman]

On 03/04/2020 10:17, Robert Scholte wrote:

This issue was registered as SUREFIRE-1768[1]
It contains a very small Maven project to demonstrate the issue.

That project contains one method executing the following:

Demo.class.getClassLoader().getResources("demo").asIterator().forEachRemaining(url 
-> {
   System.out.println(url.getFile()); // I'd like to see the 
target/classes/demo directory here at some point.
});


After executing the test it shows the following result
/E:/test-classpath-demo-master/target/test-classes/demo/
/E:/test-classpath-demo-master/target/test-classes/demo

these are similar, but more worrying: where is
/E:/test-classpath-demo-master/target/classes/demo

I rewrote it a bit by including a main method  to ensure it is not caused by 
surefire:
"%JAVA_HOME%"\bin\java --module-path target/classes --patch-module 
test.classpath.demo=target/test-classes  --module test.classpath.demo/demo.DemoTest


this gave me only one result (where I expected 2):
/E:/test-classpath-demo-master/target/test-classes/demo/


So the question is, where is
/E:/test-classpath-demo-master/target/classes/demo/

Patching is to used to replace specific classes or resources in a module 
with alternative versions. It can also be used to augment a module with 
new classes or resources. In the reproducer, it looks like module 
test.classpath.demo has been patched so that "demo" is found in the 
patch rather than the original module. This is correct behavior.


If it helps, this will give a list of the resources in the module so you 
can see the effect of the patching:


    String name = this.getClass().getModule().getName();
    ModuleLayer.boot()
    .configuration()
    .findModule(name)
    .orElseThrow()
    .reference()
    .open()
    .list()
    .forEach(System.out::println)

-Alan

Re: jlink fails with "Hash of java.xyz differs to expected hash recorded in java.base"

[Alan Bateman]

On 31/03/2020 08:50, Sebastian Stenzel wrote:

:
This was my bug report, that's why I'm investigating to give the Ubuntu guys 
some tips on how they can solve it (if they caused it in the first place). ;-)

Ah, didn't see it was you :-)

I've seen a few reports of issues on Ubuntu go by (like the javac 
launcher missing from a run-time image that claims to have the 
jdk.compiler module) which hints to me that the bits in the Ubuntu 
openjdk-* packages have additional patches and/or they are processed by 
something that is changing the bits. I don't have cycles to dig into it 
so hopefully your bug report will help the maintainers to track it down.


-Alan.