Re: Compiling module descriptors is not reproducible?
On 2018-05-19T12:26:55 +0100 Alan Bateman <alan.bate...@oracle.com> wrote: > > The requires table in the Module attribute can be used by compilers to > record version information about dependences. javac does record the > version information, I don't know if there is a way to disable that. In > any case, I assume your question will be partly answered by looking at > the output of `java --list-modules` on the different builds. If you > download OpenJDK builds from jdk.java.net/10 then you should see that > the standard and JDK-specific modules all report their version string as > "10" or "10.0.1" as they have been built with configure options that > make it so. At a guess, your mail may have included the output from an > exploded (as opposed to images) build. That's a good point: What I thought I was looking at was the compiler inserting it's own version into the class file (as in "this class file was compiled by compiler version 10-internal") as opposed to the versions of the modules being inserted. Having the platform module versions present still implies non-reproducibility but perhaps that's workable in some way. On 2018-05-19T13:49:40 +0200 Remi Forax <fo...@univ-mlv.fr> wrote: > I think it's the opposite, by default the compiler add the version of the > required module, > so the AdoptOpenJDK version does the right thing, you should have the module > compiled version by default, > the real question is why the compiler of the openjdk version 10.0.1 doesn't > generate the required module version. That is another good point! This build is the one distributed on Arch Linux. This was the procedure used to build it (no patches are applied): https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/java10-openjdk#n58 -- Mark Raynsford | http://www.io7m.com
Compiling module descriptors is not reproducible?
#8 = Module #21// "org.immutables.value"
#9 = Module #22// "com.io7m.junreachable.core"
#10 = Package#23// com/io7m/jtensors/core/determinants
#11 = Package#24// com/io7m/jtensors/core/dotproducts
#12 = Package#25//
com/io7m/jtensors/core/parameterized/matrices
#13 = Package#26//
com/io7m/jtensors/core/parameterized/vectors
#14 = Package#27// com/io7m/jtensors/core/quaternions
#15 = Package#28//
com/io7m/jtensors/core/unparameterized/matrices
#16 = Package#29//
com/io7m/jtensors/core/unparameterized/vectors
#17 = Package#30// com/io7m/jtensors/core
#18 = Utf8 module-info
#19 = Utf8 com.io7m.jtensors.core
#20 = Utf8 java.base
#21 = Utf8 org.immutables.value
#22 = Utf8 com.io7m.junreachable.core
#23 = Utf8 com/io7m/jtensors/core/determinants
#24 = Utf8 com/io7m/jtensors/core/dotproducts
#25 = Utf8 com/io7m/jtensors/core/parameterized/matrices
#26 = Utf8 com/io7m/jtensors/core/parameterized/vectors
#27 = Utf8 com/io7m/jtensors/core/quaternions
#28 = Utf8 com/io7m/jtensors/core/unparameterized/matrices
#29 = Utf8 com/io7m/jtensors/core/unparameterized/vectors
#30 = Utf8 com/io7m/jtensors/core
{
}
SourceFile: "module-info.java"
Module:
#5,0// "com.io7m.jtensors.core"
#0
3 // requires
#6,8000 // "java.base" ACC_MANDATED
#7 // 10-internal
#8,40 // "org.immutables.value"
ACC_STATIC_PHASE
#0
#9,0// "com.io7m.junreachable.core"
#0
8 // exports
#10,0 //
com/io7m/jtensors/core/determinants
#11,0 //
com/io7m/jtensors/core/dotproducts
#12,0 //
com/io7m/jtensors/core/parameterized/matrices
#13,0 //
com/io7m/jtensors/core/parameterized/vectors
#14,0 //
com/io7m/jtensors/core/quaternions
#15,0 //
com/io7m/jtensors/core/unparameterized/matrices
#16,0 //
com/io7m/jtensors/core/unparameterized/vectors
#17,0 // com/io7m/jtensors/core
0 // opens
0 // uses
0 // provides
---
Note the insertion of the "10-internal" string as constant pool entry
7.
Does anyone have any idea why this would happen? I'm considering filing
a bug at the AdoptOpenJDK project, but I'd like to be sure as to
whether or not this is actually intended behaviour by newer compilers.
Inserting version information obviously breaks reproducibility, so it'd
be nice if it could be turned off!
The sources to the above module descriptor are available [2] if anyone
wants to try compiling this for themselves. The SHA256 of the resulting
module-info.class should be
e7a94b5a2788a3c5cd6d7f586e70a0f2138a2eaa0e75144b6a16e75c4b297870 if
your compiler produces the same output as mine.
[0] https://reproducible-builds.org/
[1] https://adoptopenjdk.net/
[2] https://github.com/io7m/jtensors
--
Mark Raynsford | http://www.io7m.com
Re: JMOD, native libraries and the packaging of JavaFX
On 2018-05-09T18:53:32 +0100 Mark Raynsford <org.open...@io7m.com> wrote: > On 2018-05-07T04:19:55 -0700 > Mike Hearn <m...@plan99.net> wrote: > > > I did a bit of experimentation to learn how different operating systems > > support loading shared libraries in-memory. I also did a bit of thinking on > > the topic of "native classloaders". Here's a braindump, which may lead > > nowhere but at least it'll be written down. > > How do the BSDs cope with this? > > I suspect that OpenBSD will not have any support for this at all, but > FreeBSD might. Apologies for the cross-post, I didn't realize both addresses were on the reply. -- Mark Raynsford | http://www.io7m.com
Re: JMOD, native libraries and the packaging of JavaFX
On 2018-05-07T04:19:55 -0700 Mike Hearn <m...@plan99.net> wrote: > I did a bit of experimentation to learn how different operating systems > support loading shared libraries in-memory. I also did a bit of thinking on > the topic of "native classloaders". Here's a braindump, which may lead > nowhere but at least it'll be written down. How do the BSDs cope with this? I suspect that OpenBSD will not have any support for this at all, but FreeBSD might. -- Mark Raynsford | http://www.io7m.com
Re: ClassLoader.getResources(String)
On 2018-03-07T15:14:11 + Stephen Colebourne <scolebou...@joda.org> wrote: > > Effectively what is needed is another way for a library to be informed > of the presence of the calling application. One possible solution to > this would be to allow users to write module initialization code in > module-info.java. Then an application coder would have a solid > reliable place to put code that registers the additional configuration > files with the low-level library. There's a pattern I've seen used in OSGi that I've considered adapting for my own use (exposing resource-only modules that don't contain any code and yet need to be able to cause some other piece of code to instantiate services on behalf of the module): They call it the "extender pattern". https://dzone.com/articles/osgi-42-extender-pattern-and Briefly, what happens is that you subscribe to an interface that tells you when bundles (OSGi terminology for artifacts containing modules, more or less) are added to or removed from the system. In Jigsaw, this would probably equate to publishing some sort of event that can be observed whenever someone creates a new module layer. When you get notified that a bundle has been added, you can scan the manifest of the bundle (via the standard jar manifest APIs if you like) and can then, for example, look for manifest fields that tell you where in the jar file to find application-specific config files. X-My-Extra-Config-File: /com/example/config.xml The listening party can then read the config file, instantiate services as necessary, etc. It'd need to be handled in a way that ensured that you don't essentially race the module resolution code; Just because your code didn't get a chance to subscribe to "module became available" events until after they'd appeared doesn't mean you should miss the events. OSGi does handle this (it's written into the spec), but I don't know quite how the implementations handle it. -- Mark Raynsford | http://www.io7m.com
Re: Signing jlink code for macOS on other platforms
On 2018-02-12T11:44:07 + Mark Raynsford <org.open...@io7m.com> wrote: > Hello! > > As a long time Java developer, I've only ever had to deal with signing > jar files. I can obviously sign jar files once on whatever platform I > choose to use to build the code, and then distribute the jars to all > platforms. Build once, run everywhere, etc. > > However, now that jlink exists, as a developer I have to deal with > signing platform-specific executables. For example, if I distribute a > macOS application produced with jlink, that executable will produce a > large warning message: Seeing as I've had no response, is it safe to assume that this subject isn't permitted here? I was hoping there'd be some discussion of platform-independent ways to produce correctly-signed jlink distributions... Right now, just using jlink at all means you (probably) have to go back to building code on each individual platform in order to get access to proprietary platform-specific signing tools. This kind of reduces the utility of being able to specify another platform's JVM with --module-path, because the resulting distribution won't really be usable thanks to the erosion of the ability to run unsigned binaries on some platforms. -- Mark Raynsford | http://www.io7m.com
Re: Finding module-info.class without loading a jar
On 2018-02-24T07:53:16 +
Alan Bateman <alan.bate...@oracle.com> wrote:
>
> The JarFile API does this for you. Are you sure you've used the
> constructor that specifies the runtime version? Once you do that then
> getJarEntry("module-info.class") will locate the module-info.class in
> the versioned section. Use JarEntry::getRealName to satisfy yourself
> that it always locates the right one (say where you have a
> module-info.class in the top-level directory and one each in
> META-INF/versions/9 and META-INF/versions/10).
Ah, thank you. I had completely overlooked this constructor.
> ModuleDescriptor.read is an easy way to parse the module-info.class in
> case you need it.
Yes, I'm planning to move to this from ASM.
--
Mark Raynsford | http://www.io7m.com
Signing jlink code for macOS on other platforms
Hello! As a long time Java developer, I've only ever had to deal with signing jar files. I can obviously sign jar files once on whatever platform I choose to use to build the code, and then distribute the jars to all platforms. Build once, run everywhere, etc. However, now that jlink exists, as a developer I have to deal with signing platform-specific executables. For example, if I distribute a macOS application produced with jlink, that executable will produce a large warning message: http://ataxia.io7m.com/2018/02/12/warning.png Now obviously in the past, the system JRE was signed and so I'd give my users a jar file, they'd run the jar file using the signed JRE, and everything would work. With jlink, it's now my responsibility to sign the executables I produce. The code signing tools for macOS are evidently not available for any platform other than macOS, meaning that I now can't just build the code for all platforms on Linux if I want to use jlink (even though jlink is capable of producing embedded JREs for all of the platforms I want to support); at least part of the build would have to take place on macOS to sign the final result. This is pretty awful! Are there any plans to implement anything that's capable of signing macOS binaries and resources in a platform-independent way so that jlink-produced distributions can work without warnings? Is that even a reasonable thing to request? I've no idea how "private" Apple keep their signing implementation. -- Mark Raynsford | http://www.io7m.com
Re: Module resource as seekable byte channel?
On 2018-02-04T20:41:48 + Alan Bateman <alan.bate...@oracle.com> wrote: > > The only support for random access is in the ModuleReader API where the > read method returns a ByteBuffer containing the resource. This is very > efficient for resources in modules that are linked into the runtime > image as it's essentially a slice of a memory mapped region. It's not > efficient for modules that are packaged as JAR files as random access > poses challenges there (esp. when entries in JAR file are compressed). OK, thanks, that's what I suspected. I decided I could probably get away with simply being able to restart streams rather than seeking backwards. Implementing this turned out to be as trivial as opening the resource twice yielding streams A and B, playing from stream A until EOF, and then playing from stream B (whilst opening stream C in the background to act as the new stream A when B reaches EOF). It uses a constant amount of storage (the fixed-size read buffer per-stream) and doesn't seem to consume any extra file descriptors (because the JVM is obviously just reading bytes from a ZipEntry that it already has open behind the scenes). -- Mark Raynsford | http://www.io7m.com
Module resource as seekable byte channel?
Hello. I'm working on a problem that would seem to require accessing a (large) resource included inside a module in random manner. More specifically, I have some large audio resources inside a module and need to be able to get something analogous to a SeekableByteChannel for a given resource so that I can stream the audio and seek forwards/backwards. I could unpack the resources to the filesystem, but I'd prefer not to do this (because that would then mean I'm using double the amount of storage). Is there some way to get access to a module resource as a SeekableByteChannel in the standard library? -- Mark Raynsford | http://www.io7m.com
Re: Whitelisting modules in layers
On 2017-12-02T09:32:37 + Alan Bateman <alan.bate...@oracle.com> wrote: > > It looks like this demo is attempting to create a module layer from a > configuration that contains java.base. The java.base module is in the > boot layer, alternative implementations of java.base are not allowed in > other layers (java.base is special in this regard and the > defineModulesXXX methods are all specified to throw > LayerInstaniationException if the configuration contains java.base). Right, got it. I thought I was creating a layer that delegated to the boot layer for java.base, but obviously not. > On your question then the module layer API doesn't provide a way to > "hide" modules in the parent layers. You can override any module (except > java.base) with an alternative implementation in the child layer but it > doesn't provide anyway to hide modules that you aren't overriding. OK, thanks! It occurs to me that I'm essentially trying to do the work that the SecurityManager would have done in the past: Restrict access to packages (or in this case, entire modules). -- Mark Raynsford | http://www.io7m.com
Whitelisting modules in layers
Hello.
Is it possible to whitelist the modules available to a given layer?
I'm in the process of implementing a small experimental language that's
designed to be embedded in Java programs in a similar manner to the way
that Lua is embedded in C++ programs. The language has a module system
that I intend to map directly to Jigsaw/JPMS modules. The further
intention is that user programs are placed into their own module layer,
distinct from the host application layer (and that of the compiler).
I'd like to allow for whitelisting the modules available to user code.
Something like:
Evaluator e =
Evaluator.builder()
.setModuleSearchPath(dir1, dir2, dir3)
.setModuleWhitelist("java.base","com.example.m1","com.example.m2")
.build();
e.evaluate("[print \"hello\"]");
The user code passed to e.evaluate() would be compiled in its own module
and would only be able to create "reads" edges to java.base,
com.example.m1, and com.example.m2. Any other modules that happen to be
available to the host application would be invisible to the user's code.
I've played around with the ModuleFinder API a little, and came up
with the following:
https://github.com/io7m/moduledemo-20171201/blob/master/src/main/java/com/io7m/moduledemo/WhitelistModuleDemo.java
However, the boot_layer.defineModulesWithOneLoader() call raises an
exception: "Class loader must be the boot class loader". Clearly I'm
not meant to use the API in this way!
--
Mark Raynsford | http://www.io7m.com
Re: Retrieving the bytes of platform classes
On 2017-10-23T11:00:22 -0700 mark.reinh...@oracle.com wrote: > 2017/10/23 10:55:15 -0700, alan.bate...@oracle.com: > > Have you looked at the jrt file system provider? Details in JEP 261 [1]. > > See also JEP 220: > > > http://openjdk.java.net/jeps/220#New-URI-scheme-for-naming-stored-modules-classes-and-resources Ah, thanks! I hadn't seen this. Looks like exactly what I need. -- Mark Raynsford | http://www.io7m.com
Retrieving the bytes of platform classes
Hello. I'm writing a small tool to analyze the APIs of Java 9 modules. I'm analyzing the bytecode statically with ASM and have run into a slight problem: In order to properly analyze a class, I also need to analyze the superclasses and, in some cases, the superclasses can be platform classes. Given a platform class name such as "java/io/InputStream", is it possible to retrieve the raw bytes for the named class in a portable (read: non-platform-specific, non-JVM-specific) manner? In the olden days, we'd probably have gone rummaging in rt.jar, but that was obviously a hack and wasn't future-proof. My tool is only designed to work with JDK 9 and up. Note that I'm *not* asking to get from loaded Class instances to bytes: I'm analyzing classes statically and none of the analyzed classes will actually be loaded by any ClassLoader. -- Mark Raynsford | http://www.io7m.com
