Re: ClassLoader.getResources vs Module.getResourceAsStream

[Stephen Colebourne]

In my experience (as I've written before), ClassLoader.getResources is
perhaps the biggest pain point I've experienced in trying to move
beyond Java 8. The method seems to have been very widely used, and IMO
was considered to be preferred over Class.getResourceXxx. And it is
very confusing to use once modules come into play.

There is no simple replacement for the ability to search for resources
across jar files. Module.getResourceXxx cannot be used as it only
returns one resource from the unamed module, when there could be many
(and the docs are no particularly clear on what they do). This is very
inconvenient. Returning a stream instead of a URL also typically
involves wider code change to adopt. In addition, many projects are
still on Java 8, so can't use java.lang.Module anyway.

ServiceLoader is completely the wrong solution for config files. Its
far too heavyweight.

My solution (which I think is pretty horrible) has been to move config
files to be under META-INF.

Old location:
 org/joda/convert
New location:
 META-INF/org/joda/convert

Its backwards incompatible to downstream users, but at least it works
with ClassLoader.getResources

Given there are no good solutions to normal coding problems, I can't
help feeling that Jigsaw didn't get resource access quite right.

Stephen


On 14 July 2018 at 16:31, Alan Bateman  wrote:
> On 14/07/2018 14:00, Michał Zegan wrote:
>>
>> Hello.
>> When reading docs for jdk9 and jdk10 it seems that those methods work in
>> a bit different way when it goes to encapsulation:
>> Module.getResourceAsStream will retrieve the resource without a problem
>> if a package is opened to the caller module, probably including the fact
>> that it will find a resource when the calling module is the same as one
>> represented by the module object.
>> But, ClassLoader.getResources and other resource methods seem to require
>> unconditional package open.
>> Why? I don't quite understand that distinction.
>
> ClassLoaders, especially in a delegation chain, have no notion of "who" is
> trying to locate the resource. The ClassLoader.getResourceXXX methods are
> also not final. All told, the ClassLoader.getResourceXXX cannot reliably
> support qualified opens so this is why they are specified to only locate
> resources in modules when the package is open to all modules.
>
> The general guideline is to use Class or Module getResourceXXX when you want
> to locate a resource in your own module or another specific module. Use
> ClassLoader.getResourceXXX when you want to search the class path. If you
> follow that then it makes it a lot easier to migrate existing code to
> modules.
>
> -Alan

Re: Avoiding sun.misc.Unsafe and embracing modules in Java libraries: missing links

[Stephen Colebourne]

On 9 April 2018 at 08:33, Alan Bateman  wrote:
>> 2. Java proxies cannot invoke default methods of proxied interfaces
>>
>> The Java proxy API does not currently allow the invocation of an
>> overridden
>> default method since
>> the InvocationHandler API only supplies an instance of
>> java.lang.reflection.Method.
>
> The issue of Proxies and default methods has come up on core-libs-dev a few
> times.

I hit this problem just today but am on Java 8, so Java 9 solutions
are no good. The standard Java 8 workaround requires setAccessible. I
found an alternative that is useful in some cases, but was worth
documenting (as I didn't find it described anywhere else):
https://stackoverflow.com/a/49730826/38896

I do think Proxy should be enhanced to handle default methods without
use of MethodHandle. Proxy is more of an entry level tool, whereas MH
is lower level. Requiring MH to solve default methods seems wrong for
the typical user of Proxy.

Stephen

Re: The baby and the bathwater

[Stephen Colebourne]

On 28 March 2018 at 08:28, Peter Levart  wrote:
> That's easy to enforce in runtime. Just take a "victim" class from your
> library that is most often needed when your library is being used (or take a
> couple of them) and add a class initialization block like the following to
> them:
>
> public class Whatever {
>
> static {
> if (Whatever.class.getModule().getName() == null) {
> throw new Error("Can only use this library as a module");
> }
> }

Agreed that this has always been possible, but it is code not
metadata. Really, it should be a startup JPMS error if the module
isn't running in the expected mode. That way tools like Maven and
Gradle can also take decisions based on the metadata.

Stephen

Re: The baby and the bathwater

[Stephen Colebourne]

On 26 March 2018 at 19:08,   wrote:
> Stephen Colebourne's recent blog entry

Thanks for the thoughtful reply, of which I agree with much of it.

> Stephen's main complaint here is only about the need
> to test a library on both the class path and the module path if it's
> intended to work on Java 9 or later.  With automated testing this
> shouldn't, in principle, be a huge burden,

To a degree this depends on the size of your test suite. Some suites
are large, and running the entire suite twice in continuous
integration could be onerous.

I think the main complaint however is more subtle than a need to test
twice. It is the need to test twice _forevermore_.

ie. if this were just a transition phase, which would pass when Java
11 is the baseline, the situation would be painful but manageable. But
as things stand, there is no future time when my module will be
guaranteed to be treated as a module.

> Stephen closes with a specific suggestion:
> [snip]
> Yes, this would eliminate the need for dual testing, but only if you're
> willing to baseline to Java 11.

And that is exactly the point. At some point, Java 11 will be the new
baseline. The trade-offs jigsaw chose for the Java 8 to 9 transition
will at some point not be the right ones for the long term. There has
to be a time when a library developer can rely on strong encapsulation
and when the class-path can't just be used to completely nullify
module-info.java. Otherwise, whats the point in modularisation?

I'm arguing that it should be the module authors choice to apply the
tougher rules, but I'm very happy to hear other alternatives in the
problem-space.

> There's also at least one improvement in the JDK worth
> considering, which a few of us have already discussed, namely a small
> enhancement to javac that would allow a single invocation to compile a
> module, in module mode [2], but target class files other than
> `module-info.class` to an earlier release [3].

+1

Stephen

Re: Some points on JPMS/Jigsaw after he modularised some code (from Stephen Colebourne)

[Stephen Colebourne]

On 23 March 2018 at 13:17, David Lloyd <david.ll...@redhat.com> wrote:
> On Fri, Mar 23, 2018 at 7:51 AM, Stephen Colebourne
> <scolebou...@joda.org> wrote:
>> One approach is to say that modular jar files are always treated as
>> named modules:
>>
>> - a modular jar on the class-path is treated as being named, not part
>> of the unnamed module It is therefore encapsulated, but depends on the
>> unnamed module (class-path), so does not have full reliable
>> configuration. There is the potential for some incompatibility with
>> this change where code that uses the modular jar now can't access the
>> encapsulated packages, but this is a Good thing (as the library author
>> specifically coded for that encapsulation). Any incompatibilities are
>> smaller than JPMS has already caused, and could be managed with the
>> existing command line flags. I would hope this does not require a JVM
>> spec change to achieve.
>
> This would cause problems for containers which do not use JPMS yet
> (which is to say: containers); such frameworks may behave poorly
> (having an expectation that they were loaded as JPMS modules) or fail
> to load (if there is some kind of hypothetical enforcement at a JVM
> level).

The classes in the modular jar file would still be in the same
classloader (because same classloader module loading is the standard
strategy). Other than encapsulating internal packages, the container
really shouldn't see any changes should it? At some point containers
will support JPMS anyway.

Stephen

Re: Some points on JPMS/Jigsaw after he modularised some code (from Stephen Colebourne)

[Stephen Colebourne]

On 23 March 2018 at 10:19, Martijn Verburg  wrote:
> Stephen's comments are in his blog post:
> http://blog.joda.org/2018/03/jpms-negative-benefits.html

Firstly, I want to emphasise that my goal is effectively that of a
retrospective, to examine what hasn't worked so well and to improve
things from where we are. The jigsaw team did a great job in getting
the feature out at all - I want to make sure it is used rather than
ignored or used as a blocker to progressing the JDK.


> @Stephen are you able to share the source code?

The source code is all public:
https://github.com/ThreeTen/threeten-extra
https://github.com/JodaOrg/joda-parent
https://github.com/JodaOrg/joda-convert
https://github.com/JodaOrg/joda-beans

While some of the problems are tool-based, the most fundamental issues
are about JPMS itself.

At the heart of the problem is the split between class-path and
module-path. Since this split has happened now, and can't "unhappen"
what is needed is a way to make it easier to manage (fix the problems
the split has created).

As it stands, a library developer cannot control whether they run on
the class-path or module-path. This increases the bug-surface of the
library, and requires testing in two different environments (which is
not widely known). Without being able to insist that a library is on
the module-path it is also clear that the benefits of strong
encapsulation and reliable configuration don't apply to library
consumers.

My belief is that a way needs to be found for a library author to
insist that their library is run as a named module. There are probably
a number of ways this could be achieved - I'm interested in whether
the change makes things better, not what the specific change is.

One approach is to say that modular jar files are always treated as
named modules:

- a modular jar on the class-path is treated as being named, not part
of the unnamed module It is therefore encapsulated, but depends on the
unnamed module (class-path), so does not have full reliable
configuration. There is the potential for some incompatibility with
this change where code that uses the modular jar now can't access the
encapsulated packages, but this is a Good thing (as the library author
specifically coded for that encapsulation). Any incompatibilities are
smaller than JPMS has already caused, and could be managed with the
existing command line flags. I would hope this does not require a JVM
spec change to achieve.

A second approach is to say that the module author must mark modules strict:

- a module author could mark a module as "strict" in module-info.java
so that it is not permitted to be on the class-path. This does not
force the whole application to move to the module-path - only strict
modules and their dependencies would need to move. At runtime, if a
strict module is found on the class-path an error occurs. I suspect
this requires a JVM spec change, so may be harder.

There may well be more approaches. My gut feeling is that the first
approach is the better one, being relatively simple and implementable,
but of course I may be wrong.

Ultimately, the requirement is that library authors who go to the
effort of adding module-info.java should see some benefits in doing
so, where today they only have increased cost through running as both
an unnamed and named module.

thanks
Stephen

Re: ClassLoader.getResources(String)

[Stephen Colebourne]

On 7 March 2018 at 12:59, Alan Bateman  wrote:
> You've dismissed services but I would expect it to provide a nice solution.
> The service interface might be very simple, something like:
>
> public interface LeapSecondDataProvider {
> LeapSecondData data();
> }

Configuration and code are two very different things. Asking projects
and end users to write code for something that should be config is a
huge no-no.

My view is that JPMS has made using configuration files, especially
for libraries, a lot harder. This is a step back in usability. Just so
we are clear, for leap seconds I will now have to ask users to
manually register them using an API where previously they just added a
file. But for OpenGamma Strata, the configuration files are much more
complex and certainly unsuited to be code, even if the backwards
compatibility issues were acceptable. (This is a pattern I've used for
configuration for many years)

Effectively what is needed is another way for a library to be informed
of the presence of the calling application. One possible solution to
this would be to allow users to write module initialization code in
module-info.java. Then an application coder would have a solid
reliable place to put code that registers the additional configuration
files with the low-level library. Something like:

module com.foo.app {
  requires org.threeten.extra;

  init(ModuleInitContext context) {
UtcRules.registerLeapSecondFile("/com/foo/app/LeapSeconds.txt");
  }
}

PS. ServiceLoader is a pain to use in Java 9 too. As a library doesn't
know whether it will run as a named module or on the classpath, I have
to duplicate the service loader configuration - once in
META-INF/services and once in module-info.java, which is horrible. It
also means that the provide() static method is a feature that cannot
be used by libraries.

Stephen

Re: java.beans package in java.desktop module

[Stephen Colebourne]

What is needed is an abstraction that can work for all sorts of
data-like classes

- classic JavaBeans
- records
- value types
- HashMaps
- JSON objects
- etc

Its not rocket science as an API, but has been needed for many years
as so many projects have this code duplicated (leading to lots of
subtle differences).

The API cannot be based on method handles, as in the HashMap case
there is no property-specific method to call. But there is no reason
why the implementation of the interface for records could not use
method handles internally.

Stephen


On 7 March 2018 at 11:21, Remi Forax <fo...@univ-mlv.fr> wrote:
> As Stephen said, with the introduction of the Pattern Matching in the near 
> future, an API to extract the values from an object (the de-constructor API) 
> or at least from a record object will have to be created, but it may be based 
> on method handles, so perhaps not using a direct interface.
>
> Rémi
>
> - Mail original -
>> De: "Guillaume Smet" <guilla...@hibernate.org>
>> À: "Stephen Colebourne" <scolebou...@joda.org>
>> Cc: "jigsaw-dev" <jigsaw-dev@openjdk.java.net>
>> Envoyé: Mardi 6 Mars 2018 18:45:21
>> Objet: Re: java.beans package in java.desktop module
>
>> Hi Stephen,
>>
>> On Tue, Mar 6, 2018 at 3:29 PM, Stephen Colebourne <scolebou...@joda.org>
>> wrote:
>>
>>> It had been my hope that we might see a replacement for the java.beans
>>> package. I drew up a rough prototype here:
>>>  https://github.com/jodastephen/property-alliance
>>> based on previous work in Joda-Beans. More info here:
>>>  http://jodastephen.github.io/property-alliance/
>>>
>>> If an interface-based design were adopted, it could be implemented by
>>> the existing JavaBeans code and also by future language changes such
>>> as records (data classes), while the API could be used far more
>>> broadly, such as in Hibernate or EL.
>>>
>>> Sadly, I haven't had the time or energy to progress this, but the need is
>>> there.
>>
>>
>> Thanks for sharing. It indeed sound like something worth pursuing.
>>
>> Having a new API in the JDK indeed seems the only way out for the
>> java.beans issue.
>>
>> --
>> Guillaume

Re: ClassLoader.getResources(String)

[Stephen Colebourne]

Following up on this, it does feel like the use case is now simply not
possible. I have a similar problem with
ClassLoader.getResources(String) in threeten-extra.

https://github.com/ThreeTen/threetenbp-extra/blob/master/src/main/java/org/threeten/extra/scale/SystemUTCRules.java#L202

The ThreeTen-Extra project defines a config file
org/threeten/extra/scale/LeapSecond.txt. The code uses
ClassLoader.getResources(String) to find the latest version of the
file, which may be in the threeten-extra jar file, or in any jar file
that uses threeten-extra.jar. ie. to replace the version from
threeten-extra.jar, a user simply has to add a file with the same
name/package to their jar file.

threeten-extra.jar contains org/threeten/extra/scale/LeapSecond.txt
application.jar also contains org/threeten/extra/scale/LeapSecond.txt

Under JPMS this fails, as the resource cannot be located in
org/threeten/extra/scale in a different jar file. But this appears
makes the whole design impossible to make work with JPMS.

The code in threeten-extra.jar cannot possibly know about the package
names of the application.jar, so there is no way for it to find the
config file.

There seem to be only two solutions to this
- ServiceLoader, but that is for code, not config files
- forcing the application to manually register their config file

Both of these provide a markedly worse outcome.

Am I missing something?

Stephen


On 7 February 2018 at 20:11, Alan Bateman <alan.bate...@oracle.com> wrote:
> On 07/02/2018 16:56, Stephen Colebourne wrote:
>>
>> :
>> I was using maven to create a jar-with-dependencies file, so I could
>> use jlink. With all the code in one jar file, there shouldn't be any
>> access barriers to worry about.
>>
>> ClassLoader.getResources(String) worked just fine until Java 9. The
>> two APIs are not comparable - the ClassLoader one returns all URLs
>> found, whereas the Class one returns just one URL. Switching API would
>> change behaviour.
>
> ClassLoader.getResources searches the class path as it did in JDK 9 and
> older, it it just can't locate non-".class" resources in modules when they
> are encapsulated. Class loaders are oblivious as to who is ultimately
> attempting to load a class or locate a resource (the initiating and defining
> loader can be different, they can many class loaders in the delegation
> chain).
>
> With the uber modular JAR scenario then all classes for several libraries
> are in the same module. This means that the names of resources in that
> module are unique. If several libraries have the same resource then I assume
> you drop all but one when you create this uber JAR (or maybe you are merging
> some of the configuration files, I can't tell). So I assume you could change
> this code to use Class.getResource and it will locate at-most-one resource
> with a specific name.
>
> To do a proper migration means re-examining ResourceConfig of course. Using
> services is likely to be a lot cleaner and more robust than scanning for
> configuration files.
>
> -Alan

Re: java.beans package in java.desktop module

[Stephen Colebourne]

It had been my hope that we might see a replacement for the java.beans
package. I drew up a rough prototype here:
 https://github.com/jodastephen/property-alliance
based on previous work in Joda-Beans. More info here:
 http://jodastephen.github.io/property-alliance/

If an interface-based design were adopted, it could be implemented by
the existing JavaBeans code and also by future language changes such
as records (data classes), while the API could be used far more
broadly, such as in Hibernate or EL.

Sadly, I haven't had the time or energy to progress this, but the need is there.

Stephen


On 6 March 2018 at 13:52, Guillaume Smet  wrote:
> The java.beans package is part of the java.desktop module which is a bit
> unfortunate as the package contains quite a few classes useful to
> manipulate beans and dragging all the desktop classes with them is far from
> ideal.
>
> Typically, we have:
> - javax.el which uses java.beans.FeatureDescriptor (javax.el is the
> standard EL implementation used by Bean Validation)
> - Hibernate ORM which uses java.beans.Introspector (and thus BeanInfo and
> so on)
> - 
>
> Is there a plan to get java.beans out of java.desktop? Or should we avoid
> its usage?
>
> In our case, we can deal with the Hibernate ORM part but, for javax.el, it
> might be a bit more complicated as FeatureDescriptor is unfortunately
> included in specified APIs.
>
> Thanks for the feedback.

Re: ClassLoader.getResources(String)

[Stephen Colebourne]

On 7 February 2018 at 16:35, Alan Bateman <alan.bate...@oracle.com> wrote:
> On 07/02/2018 14:23, Stephen Colebourne wrote:
>>
>> I've been trying to use ClassLoader.getResources(String). The entire
>> application is in one named module, this includes the code that
>> invokes the ClassLoader method and the resource that it is trying to
>> find.
>
> Can you summarize what you are trying to do? If this is code in a module
> trying to locate one of its own resources then Class.getResourceXXX or
> Module.getResourcAsStream are the candidate APIs to use (not
> ClassLoader.getResourceXXX as that can never locate resources that are
> encapsulated).

I was using maven to create a jar-with-dependencies file, so I could
use jlink. With all the code in one jar file, there shouldn't be any
access barriers to worry about.

ClassLoader.getResources(String) worked just fine until Java 9. The
two APIs are not comparable - the ClassLoader one returns all URLs
found, whereas the Class one returns just one URL. Switching API would
change behaviour.

The code can be seen here:
https://github.com/OpenGamma/Strata/blob/master/modules/collect/src/main/java/com/opengamma/strata/collect/io/ResourceConfig.java#L242
It is a core part of the system that loads configuration at startup.

thanks
Stephen

ClassLoader.getResources(String)

[Stephen Colebourne]

I've been trying to use ClassLoader.getResources(String). The entire
application is in one named module, this includes the code that
invokes the ClassLoader method and the resource that it is trying to
find. The Javadoc says:

"Resources in named modules are subject to the encapsulation rules
specified by Module.getResourceAsStream. Additionally, and except for
the special case where the resource has a name ending with ".class",
this method will only find resources in packages of named modules when
the package is opened unconditionally (even if the caller of this
method is in the same module as the resource)."
https://docs.oracle.com/javase/9/docs/api/java/lang/ClassLoader.html#getResources-java.lang.String-

The call to ClassLoader.getResources(String) does not find the
resource. I assume that this is because of the last clause in the spec
"even if the caller of this method is in the same module as the
resource".

But I can't for the life of me think why such a difficult to meet
restriction has been added. The only way around it is to make the
package open, which is far from ideal. If its all within one module,
applying an access restriction like this is just unhelpful.

Stephen

Re: Module naming for logging implementations

[Stephen Colebourne]

On 26 October 2017 at 23:35, Cédric Champeau  wrote:
> There's a good argument for 1, though. log4j typically doesn't separate api
> and implementation. So a module would "require 'log4j'". It means that if
> another module like slf4j doesn't "pretend to be" log4j, it's now going to
> fail. Said differently, Jigsaw kills module replacements and fatjars.

Module replacements like log4j v1 work exactly as today. There is one
extra requirement that as well as having the same package name, the
replacement must have the same module name.

Fatjars are certainly of no use as a dependency with the current
jigsaw design, but logging doesn't seem to use fatjars.

Stephen

Re: Module naming for logging implementations

[Stephen Colebourne]

Thanks Alex and David, those are good arguments. So option 2 it is.
Stephen

On 26 October 2017 at 23:05, David Lloyd <david.ll...@redhat.com> wrote:
> On Thu, Oct 26, 2017 at 3:03 PM, Stephen Colebourne
> <scolebou...@joda.org> wrote:
>> For most service providers, option 2 is obvious, however for logging
>> it is generally the case that only one implementation should be
>> present. If all the jar files that implement a specific logging API
>> had the same module name (option 1) then the module system could
>> ensure that only one was loaded. This is a particular concern as it is
>> not uncommon for a jar file in Maven Central to depend on a specific
>> implementation of logging when it should only be depending on the API.
>
> This is actually a general special case (if you follow my meaning):
> that of an API which uses services to locate an implementation, yet
> typically only one implementation should be existent and used.  For
> example, I/O or other platform-dependent systems also spring to mind:
> they have a common API, but you generally only want one implementation
> (except maybe in some special cases - just like for logging).
>
> I think that in any case, be it a singleton-ish provider case like one
> of the above or a multiple-provider case, it is always appropriate to
> choose a unique module name for implementations. It's not appropriate
> for an API to prescribe an implementation module name.
>
> Perhaps we can enhance Jigsaw in the future with some kind of feature
> that enforces singleton implementations, if it becomes any sort of
> practical problem.
>
> --
> - DML

Fwd: Module naming for logging implementations

[Stephen Colebourne]

(previously posted on core-libs-dev, moved by request)

I've spent some time discussing module names for logging
implementations recently:
https://github.com/jodastephen/jpms-module-names/wiki/Logging-APIs
https://issues.apache.org/jira/browse/LOG4J2-2056
https://jira.qos.ch/browse/SLF4J-407?jql=text%20~%20%22jpms%22

Most logging projects are split in two - an API and an Implementation
- where the recommended solution going forwards is to use
ServiceLoader to find the implementations. A few old projects don't
have this split, and have API and Implementation together (eg
Commons-Logging).

Everyone agrees that the module name for the API must always be the
same. ie. if the SLF4J team provides a module that simulates the
Commons-Logging then it must have the same module name as
Commons-Logging.

However, there are two choices for the implementation jars.

Option 1:
All modules that implement a particular logging API must have the same
module name
eg. every module that implements "org.slf4j" (the API) must be named
"org.slf4j.impl"

Option 2:
The module name of the implementation can be whatever name makes sense.


For most service providers, option 2 is obvious, however for logging
it is generally the case that only one implementation should be
present. If all the jar files that implement a specific logging API
had the same module name (option 1) then the module system could
ensure that only one was loaded. This is a particular concern as it is
not uncommon for a jar file in Maven Central to depend on a specific
implementation of logging when it should only be depending on the API.


I'm leaning towards option 2, as it is simpler and does not require
all implementations to have the same module name (which would be
difficult to require). Any other considerations I'm missing?

Stephen

Re: java.lang.annotation.Generated

[Stephen Colebourne]

On 20 September 2017 at 14:22, Alan Bateman  wrote:
> That was the original suggestion but that package is more for annotations
> types that are used as meta annotations (@Native should have gone elsewhere
> but we can't change that now).

A point acknowledged by
http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-February/011365.html
and yet it still accepted it as the best location!

Basically, by not being in java.base it isn't useful to the people who
need it. No-one is going to pull in the java.compiler module just to
get this annotation. Jut like the original thread, I think the way
that this is used has been lost.

Stephen

Re: java.lang.annotation.Generated

[Stephen Colebourne]

Ouch. Thats an unpleasant result. It should have gone in `java.lang.annotation`.
Stephen

On 20 September 2017 at 13:49, Alan Bateman <alan.bate...@oracle.com> wrote:
> On 20/09/2017 13:43, Stephen Colebourne wrote:
>>
>> As per this email:
>>
>> http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-February/011365.html
>> the idea was to add a new annotation `java.lang.annotation.Generated`
>> to replace the old problematic one.
>>
>> Is it my imagination, or did this get forgotten?:
>>
>
> The discussion/review moved to core-libs-dev and compiler-dev in March, here
> it is:
>
> http://download.java.net/java/jdk9/docs/api/javax/annotation/processing/Generated.html

java.lang.annotation.Generated

[Stephen Colebourne]

As per this email:
 http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-February/011365.html
the idea was to add a new annotation `java.lang.annotation.Generated`
to replace the old problematic one.

Is it my imagination, or did this get forgotten?:
 
http://download.java.net/java/jdk9/docs/api/java/lang/annotation/package-summary.html

Stephen

Re: An alternative to "restricted keywords" + helping automatic modules

[Stephen Colebourne]

I don't support the ^ element or escaping like that either.

However, would adding a "module" keyword help?

module com.foo.lib {
  requires module com.foo.bar;
}

thus:

module com.foo.lib {
  requires static module blah;
  requires transitive module transitive;
}

ie. the module name is always prefixed by "module" in a "requires"
statement. But does this help?

Stephen


On 18 May 2017 at 09:59, Stephan Herrmann  wrote:
> Remi,
>
> I see your proposal as a minimal compromise, avoiding the worst
> of difficulties, but I think we can do better.
>
> Trade-off:
> In all posts I could not find a real reason against escaping,
> aside from aesthetics. I don't see this as sufficient motivation
> for a less-then-perfect solution.
>
>
> Clarity:
> I'm still not completely following your explanations, partly because
> of the jargon you are using. I'll leave it to Alex to decide if he
> likes the idea that JLS would have to explain terms like dotted
> production.
>
> Compare this to just adding a few more rules to the grammar,
> where no hand-waving is needed for an explanation.
> No, I did not say that escaping is a pervasive change.
> I never said that the grammar for ordinary compilation units
> should be changed.
> If you like we only need to extend one rule for the scope of
> modular compilation units: Identifier. It can't get simpler.
>
>
> Completeness:
> I understand you as saying, module names cannot start with
> "transitive". Mind you, that every modifier that will be added
> to the grammar for modules in the future will cause conflicts for
> names that are now legal, and you won't have a means to resolve this.
>
> By contrast, we can use the escaping approach even to solve one
> more problem that has been briefly touched on this list before:
>
> Automatic modules suffer from the fact that some artifact names may
> have Java keywords in their name, which means that these artifacts
> simply cannot be used as automatic modules, right?
> Why not apply escaping also here? *Any* dot-separated sequence
> of words could be used as module name, as long as module references
> have a means to escape any keywords in that sequence.
>
>
> Suitability for implementation:
> As said, your proposal resolves one problem, but still IDE
> functionality suffers from restricted keywords, because scanning
> and parsing need more context information than normal.
> - Recovery after a syntax error will regress.
> - Scanning arbitrary regions of code is not possible.
> Remember:
> In an IDE code with syntax errors is the norm, not an exception,
> as the IDE provides functionality to work on incomplete code.
>
>
> Stephan
>
>
> On 18.05.2017 00:34, Remi Forax wrote:
>>
>> I want to answer this before we start the meetings because i really think
>> that restricted keyword as i propose solve the issues Stephan raised.
>>
>>
>> - Mail original -
>>>
>>> De: "Stephan Herrmann" 
>>> À: jigsaw-dev@openjdk.java.net
>>> Envoyé: Mardi 16 Mai 2017 11:49:45
>>> Objet: Re: An alternative to "restricted keywords"
>>
>>
>>> Thanks, Remi, for taking this to the EG list.
>>>
>>> Some collected responses:
>>>
>>>
>>> Remi: "from the user point of view, '^' looks like a hack"
>>>
>>> This is, of course, a subjective statement. I don't share this view
>>> and in years of experience with Xtext-languages (where this concept
>>> is used by default) I never heard any user complain about this.
>>>
>>> More importantly, I hold that such aesthetic considerations are of
>>> much lesser significance than the question, whether we can explain
>>> - unambiguously explain - the concept in a few simple sentences.
>>> Explaining must be possible at two levels: in a rigorous specification
>>> and in simple words for users of the language.
>>
>>
>> I'm not against ^, or ` as it has already asked to escape an identifier,
>> but as you said it's a pervasive change that applies on the whole grammar
>> while i think that with restricted keyword (that really should be called
>> local keywords) the changes only impact the grammar that specifies a
>> module-info.java
>>
>>>
>>> Remi: "a keyword which is activated if you are at a position in the
>>>  grammar where it can be recognized".
>>>
>>> I don't think 'being at a position in the grammar' is a good way of
>>> explaining. Parsing doesn't generally have one position in a grammar,
>>> multiple productions can be active in the same parser state.
>>> Also speaking of a "loop" for modifiers seems to complicate matters
>>> more than necessary.
>>>
>>> Under these considerations I still see '^' as the clearest of all
>>> solutions. Clear as a specification, simple to explain to users.
>>
>>
>> Eclipse uses a LR parser, for a LR parser, position == dotted production
>> as i have written earlier, so no problem because it corresponds to only one
>> parser state.  Note that even if one do not use an LR or a LL parser, most
>> hand written parser i've seen, javac is one 

Re: Some suggested patches and improvements

[Stephen Colebourne]

On 12 May 2017 at 17:31, David M. Lloyd  wrote:
>>> 4. Make run-time cycle checking optional
>>
>> My opinion is that run-time cycles are inevitable. The proposed
>> solutions (refactoring to API vs Impl) is not particularly good in an
>> open source context. I'm also concerned that "requires static"
>> (optional dependencies) also naturally leads to cycles
>>
>> But the question at this point is whether it is worth the effort to
>> try and live without cycles for this release - after all, no-one wants
>> cycles in their design.
>
> It's true, cycles sound bad.  But as a point of fact, at run time they're
> not demonstrably harmful, and in fact worrying overmuch about cycles
> detracts from clean design principles: I have an ABI that conforms to a
> particular behavioral contract; it should not matter how I meet that
> contract (and in particular, whether I do so by way of a series of other
> contracts that ultimately may be used by things which use *my* ABI is a
> completely uninteresting distraction).

Just to be clear, of the five items listed, I think if just one were
to be tackled, I'd choose this one. Because I think it will come up in
real systems and because I basically agree that cycles at runtime
should not be something that should be blocked by the JDK/JVM.

Stephen

Re: Revised proposal for #AutomaticModuleNames

[Stephen Colebourne]

On 12 May 2017 at 18:29, Sander Mak  wrote:
>> module guava {
>>   requires transient com.google.common;
>> }
>>
>
> I suspect you mean 'transitive' here.

Yes :-)
Stephen

Re: Revised proposal for #AutomaticModuleNames

[Stephen Colebourne]

On 5 May 2017 at 15:41,   wrote:
> I suspect what you really mean is "Never publish JARs that refer to
> automatic modules that do not have `Automatic-Module-Name` attributes".
> In general that's good advice.  It might even be reasonable for managers
> of artifact repositories to insist upon it, although I now understand
> that that could be problematic for a repository as popular as Maven
> Central.

I think this is the tricky part. Lets review where we are.

Firstly, we now have super-package reverse-DNS names and MANIFEST
entries. Both of these are a very good thing.

Secondly, a module with a module-info can now depend on three things:
- an explicit module with a module-info  (good)
- an automatic module with MANIFEST entry  (good)
- an automatic module with name implied from filename  (not good)

Thirdly, given the nature of Maven Central, it is not realistic to
prevent modules with dependencies on names implied from filenames
getting into Central.

My remaining concerns with automatic modules are primarily about this
last point - we have to accept that there will be modules with "bad
dependencies" in Maven Central. For example, the graph will end up
with a dependency on "guava" that breaks when Guava is properly
modularized and uses "com.google.common" as the module name. The
result is a graph that does not resolve. While I and others will try
and communicate this issue to the community, I would prefer that the
issue simply could not occur.


However, it has been pointed out to me off-list that there is a way to
workaround the name change that I had not fully appreciated before.

Given this broken module graph state:

 module com.foo.application {
   requires guava;
 }
 module com.google.common {
 }

it is possible to fix it by introducing an additional module as follows:

 module guava {
   requires transient com.google.common;
 }

With this additional "renaming shim module" in the graph, everything
now works again.

Given that there is a potential workaround, I am not as concerned as I
previously was about this issue.


Ideally, the creation of this "renaming shim module" should be part of
the JDK (effectively it is a kind of alias, and the JDK is best placed
to manage this). However, it should also be possible for tools like
Maven to provide the shim dynamically when needed. It would even be
possible to release such a shim module to Maven Central and depend on
it in the usual way.

While this is still a workaround to an issue I wish didn't exist in
the first place, it is a viable solution that I think allows automatic
modules to proceed as is (given the JSR/release timeline pressure).

Stephen

Re: Some suggested patches and improvements

[Stephen Colebourne]

On 12 May 2017 at 01:43, David M. Lloyd  wrote:
> 1. Layer primitive: addExports() - mirrors the existing Module.addExports()
> method for ModuleLayer.Controllers
> 2. Layer primitive: addUses() - mirrors the existing Module.addUses() method
> for ModuleLayer.Controllers
> 3. Layer primitive: addPackage() - allows ModuleLayer.Controllers to add
> packages to a module after it has been defined

Are these a good idea? I don't know. What I do know is that the use
cases for them seem to be focussed on low-level coders and framework
writers, which is a very small subset of all Java developers, and a
group who can work around difficulties.

> 4. Make run-time cycle checking optional

My opinion is that run-time cycles are inevitable. The proposed
solutions (refactoring to API vs Impl) is not particularly good in an
open source context. I'm also concerned that "requires static"
(optional dependencies) also naturally leads to cycles

But the question at this point is whether it is worth the effort to
try and live without cycles for this release - after all, no-one wants
cycles in their design. Since there is a workaround, via the
low-level module/layer code, it feels like we should be willing to
take the punt and try to live without cycles in 9 keeping it under
review to add in 10.

> 5. Add optional class loader isolation for modules on the module path

Again, my opinion is that the isolation of modules is insufficient in
Jigsaw and that hidden packages will sometimes be very messy. However,
the only tool available to solve this appears to be classloaders, and
they have plenty of flaws themselves. While opt-in isolation appears
tempting, I'd prefer to wait and see a better solution in a later
version. Maybe the answer is to deprecated class loaders and come up
with something better!


Given where we are, I think my preference is to see a JDK 9 release
soon rather than end the JSR over these issues.

Stephen

Explicit file names in module-info - #AutomaticModuleNames

[Stephen Colebourne]

This is a proposal to mitigate some of the worst effects of automatic
modules. Partial modules remain the better solution.

We now have an agreed naming convention (reverse DNS), which is
something we haven't had before. This gives us relatively stable and
predictable names.

But we also know that the module name will differ from the filename in
most cases. As demonstrated before, having two disconnected names will
block migration when forced to move from the filename to the module
name.

Thus, the biggest design problem is that the module depending on an
automatic module is being asked to use the wrong name, with no link to
the right one which they can now predict with reasonable certainty.

As such, my proposal is that the module depending on an automatic
module must specify the long-term stable module name, with the
short-term filename included as additional metadata.

Here is a keyword approach to this:

module org.joda.convert {
  requires com.google.common file guava;
}

Here is an annotation approach that could be used instead:

@Alias(moduleName = "com.google.common" fileName="guava")
module org.joda.convert {
  requires com.google.common;
}

With both of the above, the module declaration uses the correct final
module name from day one, which is vital to not creating future
problems. The long-term stable name can be communicated in many ways,
such as on a website or by email, not just by code or manifest. I can
now release Joda-Convert, whether or not Guava has been modularised.
Success

The short-term migration-only nature of the filename is much much
clearer. The filename can be removed once the dependency is a module
(and static analysis tools could highlight that). The fileName would
be the Maven artifactId in most cases. (There is a case for the
fileName to be a list of possible filenames for additional
flexibility).

I think this design addresses some of Roberts's concern too. With this
plan, Maven Central would contain modules depending on automatic
modules, but the dependency names would be sufficiently stable for
this not to be the major issue it has been previously.

While I don't think automatic modules are the best option, were the
above chosen, I think it would be a solution the community could
successfully and easily adopt.

Stephen

Re: Revised proposal for #AutomaticModuleNames

[Stephen Colebourne]

On 4 May 2017 at 18:38,  <mark.reinh...@oracle.com> wrote:
>   - Define a JAR-file manifest attribute, `Automatic-Module-Name`, whose
> value is used as the name of the automatic module defined by that JAR
> file when it is placed on the module path, as previously suggested

This is a step forward wrt automatic modules.

> If a JAR file on the module path does not have such a
> manifest attribute then its automatic-module name is computed using
> the existing filename-based algorithm.

Which we all agree is almost guaranteed to be wrong.

>   - To increase awareness of when automatic modules are used, and of the
> consequences of their use, suggest that Java language compilers issue
> two new types of warnings, and implement these warnings in the RI:

Perhaps these should be reconsidered? If the module depends on an
automatic module with a module name in the manifest, it can be
considered reasonably stable. Whereas anything depending on a filename
derived module name is unstable. That isn't being captured by the
warnings.

> In related threads on jigsaw-dev both Stephen Colebourne [e] and Robert
> Scholte [f] suggest a new kind of explicit module, variously called
> "partial" or "incomplete" or "soft", which can read all unnamed modules
> so that dependencies that have not yet been modularized can be left on
> the class path.  This is an attractive notion on the surface, and in fact
> one that we partly implemented in the prototype code base over two years
> ago, using the term "loose".  We didn't pursue it then, however, and I
> don't think it's worth pursuing now, for two reasons:
>
>   - If "partial" (or whatever) modules were to replace automatic modules
> then application developers would have to wait until all of their
> dependencies have been at least "partially" modularized before they
> can even begin to experiment with modularizing their own code.

This seems demonstrably false.

module com.foo.application {
  requires com.foo.lib;
}
partial module com.foo.lib {
  exports com.foo.lib;
  // TODO: requires guava
  // TODO: requires joda-beans
}
modulepath = application
classpath = guava, joda-beans

The application developer can write the module declarations above, and
use them even though all the external dependencies are yet to be
modularised. It seems clear to me that the application developer has
modularised their application, even if not completely.

Given that this seems demonstrably false, I find the claim that
"application developers would have to wait" rather odd. This suggests
that something is perhaps not fully understood in the text above or
the partial proposal.

>   - We could solve that by keeping automatic modules and adding partial
> modules, but then we'd have a more complex module system and a more
> complex story to tell.

I agree with this. For me it either automatic or partial, not both.
And partial is a lot better in technical and community terms IMO as it
is a mechanism for reliable gradual migration. I still fail to see why
dumping the jars of the classpath on the modulepath and hoping it
works is a good strategy for migration - it seems like a bad short
cut.

Stephen


On 4 May 2017 at 18:39,  <mark.reinh...@oracle.com> wrote:
> Thanks to everyone, and especially Stephen Colebourne, Brian Fox, and
> Robert Scholte, for the extensive feedback.
>
>   
> http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-May/000687.html
>
> TL;DR: Keep automatic modules, bring back the module-name JAR-file
> manifest attribute, and strongly recommend reverse-DNS module names.
>
> Comments?
>
> - Mark

Re: Can automatic modules be made to work?

[Stephen Colebourne]

On 27 April 2017 at 19:50, Robert Scholte  wrote:
> The returning question is: how can I as a *library builder* participate in
> adopting Jigsaw?

Indeed, the community really wants to help JPMS adoption and add
module metadata. Since bottom up won't work, a mechanism needs to be
provided that enables migration of any jar file, whether its
dependencies have been modularized or not.

> Which made me think of the concept of soft and strict modules. Assuming
> 'strict' is the preferred default, 'soft' or any equivalent alternative
> would be a new keyword which has the same effect as add-reads
> =ALL-UNNAMED, but it's information is available at both compile time
> and runtime.
> With soft modules you can require a subset of modules.

I think that the notions expressed recently of "requires classpath",
"partial requirements", "incomplete/complete modules" [1] and
"strict/soft modules" (above) are all fundamentally the same idea.
Robert expresses a possible underlying mechanism well I think.

I'd envisage an additional keyword, or some other syntax, to mark a
module as not being able to specify all dependencies (because they are
not modules yet):

soft module org.joda.beans {
  requires org.joda.convert;
  // TODO requires guava, revisit when guava is released as a module
}

With this approach, I think JPMS adoption could be quite strong in the
community.

Stephen

[1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-April/012370.html

Can automatic modules be made to work?

[Stephen Colebourne]

On 26 April 2017 at 17:27,   wrote:
> I think I need to reconsider my previous conclusion that explicit modules
> that depend upon automatic modules should never be published for broad
> use [2].
>...
> The only remaining objection seems to be the aesthetic one, i.e., the
> fact that the name of an automatic module is derived from the artifact
> that defines it rather than from some intrinsic property of its content
> or an explicit declaration by its author.  I understand and share this
> concern.  I completely agree that modules are not artifacts, as I've
> written before [3], but I don't see a better alternative [4].

OK, so in this thread, I'll outline what changes could potentially
allow distributed modules based on automatic modules to work. This is
not an endorsement of the concept, but a pragmatic take on what could
be done now (other than delay JDK9 or remove the modulepath from
JDK9).

Some basic assertions:
1) Modules != Artifacts [1]
2) Module names should thus be aligned with code, not artifacts [2]
3) Automatic modules currently derive their name from the artifact,
which is almost always wrong wrt #2
4) A module author is forced to choose the artifact name initially,
and change it to the module name when released
5) Since multiple artifact names point represent the same module [1],
there are guaranteed to be problems in large graphs
6) There is no way out when module hell hits

To succeed with distributing modules depending on automatic modules,
it seems to me that we need:
1) a naming strategy that is reliable for the person making the guess
2) an approach for JPMS to link the guessed name to an artifact

I'd argue that the naming strategy can be relatively simple - the
highest package name [2]. Feedback has been pretty much universal in
agreeing to super-package reverse-DNS so far, thus the chances of the
guess being right are definitely increased. While not a perfect, it
might just about be good enough.

The second point, what does JPMS do, has yielded various options and
much discussion [3]. If we accept the notion that we are using
super-package reverse-DNS module names, then we can limit the options
on the table to those that produce names of that type. This implies
that the name of an automatic module is derived from the packages in
the jar file.

Since not every jar file has a single super-package, we need a four
step strategy:

1) Use Module-Name in MANIFEST.MF if present. This allows low-tech
projects to override the JPMS strategy. It would be particularly
useful for old branches of active codebases, such as commons-lang v2
or commons-collections v3 where they want to actively publish the
module name rather than leave it implied.

2) For each non-modular jar on the modulepath, consider each
super-package to be a separate module name. Since automatic modules
are open and can see each other, it does not matter if two or more
automatic modules are produced from the same jar file. This handles
most jar files on Maven Central, including odd ones like Colt where
there are multiple super-packages [4].

3) If a module name dependency is still not found, further examine the
non-modular jar files. Consider each and every package name to be a
potential module name (yes, every single package). This handles Colt
where `cern.colt` would be the most sensible module name.

4) If a module name dependency is still not found, further examine the
non-modular jar files. If a non-modular jar file has two or more
packages with the same stem but no code at the level of that shared
stem, treat the stem as a potential module name, provided it does not
clash with any other module name. For example, a jar file that has
`com.google.common.io` and `com.google.common.base` would have a
shared stem of `com.google.common` This handles Guava [5]. eg. In
Colt, `cern.jet` would be a module name by these rules.

In other words, a non-modular jar does not have one module name, it
has a number of possible module names, checked in order, for the
purpose of matching the missing module names in the module graph.

As always, the success of the approach to automatic modules will
depend on the ability of module authors to guess the name correctly,
but hopefully they will do OK, particularly if there are some
published rules, or a website suggesting the best option for a given
jar file.


As I started with, the hoops necessary to get automatic modules to
work indicate to me that they are not the right solution to the
gradual migration problem. But if they are to stay, this is my take on
what is needed.

Stephen


[1] http://blog.joda.org/2017/04/java-se-9-jpms-modules-are-not-artifacts.html
[2] http://blog.joda.org/2017/04/java-se-9-jpms-module-naming.html
[3] 
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-April/000666.html
[4] https://dst.lbl.gov/ACSSoftware/colt/api/index.html
[5] http://google.github.io/guava/releases/21.0/api/docs/

Re: Alternatives for naming automatic modules, and a proposal (#AutomaticModuleNames)

[Stephen Colebourne]

On 26 April 2017 at 19:35, Alan Bateman <alan.bate...@oracle.com> wrote:
> On 26/04/2017 18:35, Stephen Colebourne wrote:
>
>> :
>> - an incomplete module defines as many dependencies as it can
>> - an incomplete module also depends on the classpath
>> - the current automatic module concept ceases to exist
>> (an incomplete module is not an automatic module, as it doesn't get to
>> read all other modules. It only gets to read the classpath)
>>
> This is along the lines of what we've previously tossed around as `requires
> $CLASSPATH`. The JAR files on your class path may expose type in other
> libraries that get migrated to modules. In the extreme then all the
> dependences are migrated so that the class path is empty. So more
> readability is needed to avoid lock step migration (it's the same
> readability that automatic modules get today).

I'm working on the principle that the classpath sees all modules anyway.

FWIW, I'll go on record and say that I think the bifurcation between
the classpath and modulepath is a mistake that will be damaging long
term. I would prefer to see all jar files continue to sit on the
classpath, just that some also have additional metadata controlling
their exports and dependencies. The complete/incomplete module
declarations described above, would essentially allow this AFAICT.

Mark R wrote:
> This approach would, moreover, still require existing projects
> to do new releases, which as you indicate above can be very costly and
> in many cases unlikely to happen.

True, but the module metadata would always be correct no matter how
many releases are needed (much easier for Maven Central to validate
this too). Any project that adds an incomplete module declaration is
obviously still alive, so much more likely than average to finish the
process and become a complete module.

Stephen

Re: Alternatives for naming automatic modules, and a proposal (#AutomaticModuleNames)

[Stephen Colebourne]

On 26 April 2017 at 17:27,  <mark.reinh...@oracle.com> wrote:
> 2017/4/25 5:08:21 -0700, Stephen Colebourne <scolebou...@joda.org>:
>> As discussed before, removing automatic modules and allowing modules
>> to have partially specified dependencies [1] 1b allows projects to
>> migrate to modules immediately. This would be a huge win. Any project
>> that is interested could write module-info immediately, even if all it
>> was doing was specify the exported packages. They would release within
>> their normal release cycles, and depend on whatever other modules are
>> already released at the time of release. As more and more modules
>> become available, the partial requirements will turn to complete
>> requirements, and thus explicit modules.
>
> This approach is only a little better than that of the `Module-Name`
> manifest attribute.  At least it doesn't establish two different ways
> to name a module, but it still makes it easy for people other than the
> maintainer of a component to try to establish that component's module
> name.  This approach would, moreover, still require existing projects
> to do new releases, which as you indicate above can be very costly and
> in many cases unlikely to happen.

Perhaps this concept is not fully understood. The concept is:

- there are two types of module - "complete" and "incomplete"
- a module can only declare a "requires" clause on another module, ie.
one with a module declaration
- a complete module fully defines all its dependencies
- an incomplete module defines as many dependencies as it can
- an incomplete module also depends on the classpath
- the current automatic module concept ceases to exist
(an incomplete module is not an automatic module, as it doesn't get to
read all other modules. It only gets to read the classpath)

Note that at no stage does a module declaration author attempt to
guess a module name. The module declaration author is only able to
depend on things with known names.

The set of modules sits on the modulepath. The set of non-modules sits
on the classpath. The two do not mix.

Stephen
(I'll reply to some other points later when I'm not heading for the train...)


> 2017/4/25 5:08:21 -0700, Stephen Colebourne <scolebou...@joda.org>:
>> On 24 April 2017 at 19:54,  <mark.reinh...@oracle.com> wrote:
>>> An explicit module that depends upon one or more modules that are
>>> automatic today is, itself, no more stable than those automatic modules.
>>> It could be broken when those automatic modules are modularized
>>> explicitly, and if it `requires transitive` an automatic module then any
>>> modules that depend upon it could be broken when that automatic module
>>> is modularized explicitly.
>>
>> I find this to be a completely artificial pure view of the world. Most
>> projects do not have internal packages, and if they do, most end-users
>> do not use them. When developers upgrade an artifact in Maven, they
>> expect to have incompatibilities, so a smaller set of packages is just
>> fine. Its a normal part of software development today, not an
>> aberration.
>
> If developers expect to have to fix incompatibilities when they upgrade
> to a newer version of an artifact then what's wrong, after all, with
> publishing an explicit module that depends upon non-modularized
> components in the form of automatic modules?
>
> If one of those automatic modules is modularized later on, and given a
> different name, then how is having to fix that materially different from
> having to fix code that was using a package that's no longer exported?
> If anything it might actually be easier to cope with a module-name
> change, since all you have to do is edit a `requires` directive; code
> that refers to a package that's now encapsulated might require a
> non-trivial rewrite.
>
> I think I need to reconsider my previous conclusion that explicit modules
> that depend upon automatic modules should never be published for broad
> use [2].  Sure, they have unstable names and unstable APIs but if, as you
> say, people are generally used to fixing minor problems when they upgrade
> then this instability may well be tolerable -- especially since it would
> allow maintainers to modularize whenever they want to rather than only
> after all of their dependencies have been modularized.
>
>> ...
>>
>> On another thread you wrote:
>> On 24 April 2017 at 16:46,  <mark.reinh...@oracle.com> wrote:
>>> Automatic modules are best viewed as a transitional tool that allows you
>>> to modularize an isolated, unpublished set of components in a top-down
>>> fashion over time.  You can use existing published JAR files, fr

Re: Alternatives for naming automatic modules, and a proposal (#AutomaticModuleNames)

[Stephen Colebourne]

Just to add a note that came up in a twitter conversation.

Modules are not artifacts. [1]
ergo, module names are usually not artifact names
yet, automatic module names are derived from artifact names

Its no wonder that I have seen lots of confusion between modules and
artifacts, Jigsaw itself is mixing the two concepts!

Stephen

[1] http://blog.joda.org/2017/04/java-se-9-jpms-modules-are-not-artifacts.html


On 25 April 2017 at 13:08, Stephen Colebourne <scolebou...@joda.org> wrote:
> On 24 April 2017 at 19:54,  <mark.reinh...@oracle.com> wrote:
>> An explicit module that depends upon one or more modules that are
>> automatic today is, itself, no more stable than those automatic modules.
>> It could be broken when those automatic modules are modularized
>> explicitly, and if it `requires transitive` an automatic module then any
>> modules that depend upon it could be broken when that automatic module
>> is modularized explicitly.
>
> I find this to be a completely artificial pure view of the world. Most
> projects do not have internal packages, and if they do, most end-users
> do not use them. When developers upgrade an artifact in Maven, they
> expect to have incompatibilities, so a smaller set of packages is just
> fine. Its a normal part of software development today, not an
> aberration.
>
> Frankly though, if you think automatic modules are as bad as you make
> out, you should simply remove them.
>
> On another thread you wrote:
> On 24 April 2017 at 16:46,  <mark.reinh...@oracle.com> wrote:
>> Automatic modules are best viewed as a transitional tool that allows you
>> to modularize an isolated, unpublished set of components in a top-down
>> fashion over time.  You can use existing published JAR files, from Maven
>> Central or elsewhere, as automatic modules.  The names and APIs of those
>> automatic modules are unstable and will change as they are modularized
>> explicitly, but in an isolated set of components you can easily adjust
>> your own `module-info.java` files as needed to cope with those changes.
>
> And for what benefit? Why would a company put itself through the pain
> of relying on potentially thousands of external dependencies via
> automatic modules, taking a hit every time one gets converted to a
> real module. Where is the payback for that company? What is the
> benefit they gain? If this is the only use case for automatic modules,
> then they should be dropped.
>
> Unfortunately, the concept of bottom-up full modularization simply
> won't work, no matter how much the Jigsaw team hopes it will. The
> process would take forever, may not be possible for some projects,
> will be side-tracked into the release cycles of larger projects, be
> blocked by dead projects and for many other reasons just stall. I'd
> also note that everyone outside Oracle has given the same message.
>
> For example, there are 42 separate projects at Apache Commons, some of
> which have multiple release streams - thats an awful lot of work
> you're expecting unpaid volunteers to do, particularly when each has
> to be released in order. (A typical release at Apache Commons is a
> multi-week affair, so even if Apache Commons had the energy to release
> everything, it would take all their energy for well over a year.)
> There are even 11 different Joda projects to work on, with active ones
> depending on less active ones, I would be forced to do artificial
> releases just to satisfy a bottom-up migration.
>
> The only approach that makes any sense to migration is to drop the
> artificial purity goal. We already have build tools (Maven, Gradle)
> that manage versions, locking them in to form a working graph of
> artifacts. When a version is changed, sometimes things break, and we
> fix them. This is all normal software development. It is not something
> for JPMS to try and fight or fix.
>
> As discussed before, removing automatic modules and allowing modules
> to have partially specified dependencies [1] 1b allows projects to
> migrate to modules immediately. This would be a huge win. Any project
> that is interested could write module-info immediately, even if all it
> was doing was specify the exported packages. They would release within
> their normal release cycles, and depend on whatever other modules are
> already released at the time of release. As more and more modules
> become available, the partial requirements will turn to complete
> requirements, and thus explicit modules.
>
> Plus, it is really easy for the build tool to work with. If the jar
> has a module-info it goes on the modulepath, otherwise it goes on the
> classpath. And AFAICT it would just work.
>
> Worrying about getting incomplete "impure" modules in Mav

Re: Alternatives for naming automatic modules, and a proposal (#AutomaticModuleNames)

[Stephen Colebourne]

Maven Enforcer Plugin
http://maven.apache.org/enforcer/enforcer-rules/banTransitiveDependencies.html
Checks for lots of different things, including transitive deps.

Stephen

On 25 April 2017 at 17:00, Brian Fox  wrote:
> Here's one I'm familiar with:
> https://maven.apache.org/plugins/maven-dependency-plugin/analyze-mojo.html
> At least within Maven, it's a known best practice to ensure you're not
> dependent on transitives.
>
> On Tue, Apr 25, 2017 at 11:44 AM,  wrote:
>>
>> 2017/4/25 6:53:45 -0700, bri...@infinity.nu:
>> > ...
>> >
>> > While it's technically true you can consider all the exports to be part
>> > of
>> > the API, the reality is that most libraries aren't used that way. In
>> > fact,
>> > there are commonly accepted tools to detect when you are depending on a
>> > transitive dependency that isn't explicitly declared by you because it
>> > is
>> > dangerous.
>>
>> Which tools are those, and are they broadly used?
>>
>> - Mark
>
>

Re: Alternatives for naming automatic modules, and a proposal (#AutomaticModuleNames)

[Stephen Colebourne]

On 25 April 2017 at 14:31, Alan Bateman  wrote:
> Someday then we might get to the point where modules on the application
> module path could be assigned to different class loaders but it has huge
> implications and would take an entire release to shake out issues.

Has this been documented anywhere? Because all modules in one
classloader seems like entirely the wrong default and I don't
understand the trade off. Clashes on concealed packages will not be
popular.

Stephen

Re: Alternatives for naming automatic modules, and a proposal (#AutomaticModuleNames)

[Stephen Colebourne]

On 24 April 2017 at 19:54,   wrote:
> An explicit module that depends upon one or more modules that are
> automatic today is, itself, no more stable than those automatic modules.
> It could be broken when those automatic modules are modularized
> explicitly, and if it `requires transitive` an automatic module then any
> modules that depend upon it could be broken when that automatic module
> is modularized explicitly.

I find this to be a completely artificial pure view of the world. Most
projects do not have internal packages, and if they do, most end-users
do not use them. When developers upgrade an artifact in Maven, they
expect to have incompatibilities, so a smaller set of packages is just
fine. Its a normal part of software development today, not an
aberration.

Frankly though, if you think automatic modules are as bad as you make
out, you should simply remove them.

On another thread you wrote:
On 24 April 2017 at 16:46,   wrote:
> Automatic modules are best viewed as a transitional tool that allows you
> to modularize an isolated, unpublished set of components in a top-down
> fashion over time.  You can use existing published JAR files, from Maven
> Central or elsewhere, as automatic modules.  The names and APIs of those
> automatic modules are unstable and will change as they are modularized
> explicitly, but in an isolated set of components you can easily adjust
> your own `module-info.java` files as needed to cope with those changes.

And for what benefit? Why would a company put itself through the pain
of relying on potentially thousands of external dependencies via
automatic modules, taking a hit every time one gets converted to a
real module. Where is the payback for that company? What is the
benefit they gain? If this is the only use case for automatic modules,
then they should be dropped.

Unfortunately, the concept of bottom-up full modularization simply
won't work, no matter how much the Jigsaw team hopes it will. The
process would take forever, may not be possible for some projects,
will be side-tracked into the release cycles of larger projects, be
blocked by dead projects and for many other reasons just stall. I'd
also note that everyone outside Oracle has given the same message.

For example, there are 42 separate projects at Apache Commons, some of
which have multiple release streams - thats an awful lot of work
you're expecting unpaid volunteers to do, particularly when each has
to be released in order. (A typical release at Apache Commons is a
multi-week affair, so even if Apache Commons had the energy to release
everything, it would take all their energy for well over a year.)
There are even 11 different Joda projects to work on, with active ones
depending on less active ones, I would be forced to do artificial
releases just to satisfy a bottom-up migration.

The only approach that makes any sense to migration is to drop the
artificial purity goal. We already have build tools (Maven, Gradle)
that manage versions, locking them in to form a working graph of
artifacts. When a version is changed, sometimes things break, and we
fix them. This is all normal software development. It is not something
for JPMS to try and fight or fix.

As discussed before, removing automatic modules and allowing modules
to have partially specified dependencies [1] 1b allows projects to
migrate to modules immediately. This would be a huge win. Any project
that is interested could write module-info immediately, even if all it
was doing was specify the exported packages. They would release within
their normal release cycles, and depend on whatever other modules are
already released at the time of release. As more and more modules
become available, the partial requirements will turn to complete
requirements, and thus explicit modules.

Plus, it is really easy for the build tool to work with. If the jar
has a module-info it goes on the modulepath, otherwise it goes on the
classpath. And AFAICT it would just work.

Worrying about getting incomplete "impure" modules in Maven Central is
simply the wrong concern. If we keep the current design, and insist on
no automatic module dependencies in Maven Central, then JPMS will
simply not be adopted outside a few private niches.

Stephen

[1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-March/011686.html

Re: Alternatives for naming automatic modules, and a proposal (#AutomaticModuleNames)

[Stephen Colebourne]

On [1] the conclusion given the premise is not unreasonable. The file
name can be easily changed by a developer or build tool to match the
expected module name. However, it is the premise I strongly object to:

"The fundamental problem here is that we're trying to infer a .. name..."

Inferring a name (identifier) is generally a bad idea in any
programming context, but especially so in this one where it will be
baked into the compiled artifacts. Accepting that inferring a name is
a fundamentally bad idea leads to examination of alternatives to
automatic modules [3].

Such an examination yields an approach where a module only specifies
dependencies on proper modules, with some ability to express that its
dependencies are incomplete. This appears not to have been considered
despite being equivalent in aiding migration yet avoiding any need to
infer a name.

As such, on [2], I find the new warnings to be little more than a
sticking plaster on the wart of automatic modules. The need to add a
disclaimer "Strongly advise developers never to publish, for broad
use, explicit modules that require automatic modules" while a welcome
recognition of the problem, merely serves to emphasise how broken the
concept of automatic modules is. They simply should not be part of the
Java platform, a millstone around our necks forevermore.

All I hope the teams at Maven, Gradle, Sonatype, JFrog and others can
work to ensure strong validation to exclude automatic modules from
public repositories, although this is merely an attempt to lock the
door after the horse has bolted.

I also note that proposals to date do not provide strong guidance on
module naming. As things stand, there is likely to be inconsistency in
the choices made between projects. This is particularly galling given
the extremely strong guidance about NPM and the need for namespaces
[4] [5]. It is important to note the impact of what happens when
projects collide due to lack of appropriate namespacing [6] [7]. While
boring and verbose, reverse DNS is the approach that has served Java
well for 20+ years, and should be strongly promoted here.

Stephen


[1] 
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-April/000666.html
[2] 
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-April/000667.html
[3] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-March/011686.html
[4] 
http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-January/000537.html
[5] https://github.com/npm/npm/issues/798
[6] http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm
[7] 
https://www.techdirt.com/articles/20160324/17160034007/namespaces-intellectual-property-dependencies-big-giant-mess.shtml


On 3 April 2017 at 17:35,   wrote:
> Thanks for the continued feedback on this difficult issue.
>
> FYI:
>
>   
> http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-April/000666.html
>   
> http://mail.openjdk.java.net/pipermail/jpms-spec-experts/2017-April/000667.html
>
> - Mark

Re: #VersionsInModuleNames

[Stephen Colebourne]

Just to complete this thread, see:
http://mail.openjdk.java.net/pipermail/jpms-spec-observers/2017-March/000839.html
thanks for the revised proposal.
Stephen



On 15 March 2017 at 10:13, Stephen Colebourne <scolebou...@joda.org> wrote:
> Responding to the discussion on the EG list.
>
> I am now strongly of the opinion that using the language to restrict
> module names to not end in a number is a mistake. The two clear cases
> show two reasons why it is a mistake.
>
> commons-lang3 is the third version of commons-lang. It was named  this
> way because the API changed, and there was a strong desire to allow
> both commons-lang and commons-lang3 to be on the classpath. Note that
> the package for commons-lang is org.apache.commons.lang and the
> package for commons-lang3 is org.apache.commons.lang3.
>
> (Again, if we were to name modules after the highest package, it would
> be obvious what the relationship is, and why restricting numbers
> doesn't make sense, because they are allowed in package names)
>
> fabric8 is a brand name, and an example of the future of naming
> projects. Over time, more and more good project names are being used
> up (just like good user names). Roll forward 10 years, and the chances
> of finding a good project name will decrease. Using a number as part
> of the name is one way to sidestep the problem and expand the number
> of available names.
>
> If the current #VersionsInModuleNames plan is not changed, the key
> question is What Should These Projects Name Themselves?
>
> commons-lang-three?
> Maybe, but yuck.
>
> fabricate?
> Maybe, but really that is a completely different name, and the whole
> point of using the 8 was to avoid using the -ate project name (maybe
> someone else owns "fabricate").
>
> I'm sure there are plenty of other examples on Maven Central. But it
> doesn't really matter. Both of these are valid reasons to name a
> project with a number at the end. As such, the #VersionsInModuleNames
> proposal cannot stand.
>
>
> Since part of the reason for this proposal was automatic modules,
> which have caused problems in other ways, I'll suggest the following
> change to automatic modules. Automatic modules must either contain the
> Module-Name MANIFEST entry, or have a file name that exactly matches
> the desired module name. ie. the standard jar files downloaded from
> Maven Central, eg foo-bar-1.2 must be renamed to be used as an
> automatic module. This means that the proposed rules for removing the
> trailing version and converting dashes to dots would be removed from
> the spec. Build systems and developers are perfactly capable of
> renaming a file - baking specific naming conversion rules into the
> JPMS is inadvisble.
>
> (Note that the above is not an endorsement of automatic modules in
> general, I believe that there are better solutions to the migration
> problem. But the above is an imporvement to the current state of the
> JPMS.)
>
> thanks
> Stephen

Re: Better tools for adjusting to strong encapsulation

[Stephen Colebourne]

On 23 March 2017 at 03:30,   wrote:
> It appears that issuing warning messages for illegal-access operations
> enabled by the precise `--add-opens` and `--add-exports` options is a
> bit too aggressive, at least for JDK 9.  Perhaps we can enable that in
> JDK 10 after there's been more time for libraries, frameworks, and even
> the JDK itself to adjust to the realities of strong encapsulation.
>
> For now I suggest that we revert to the previous behavior of these two
> options, so that they do not cause warning messages to be issued.  The
> new `--permit-illegal-access` option will continue to generate warning
> messages, as proposed.  If those messages are a problem in a particular
> scenario then they can be eliminated by switching to an appropriate set
> of `--add-opens` options, which can be constructed from the information
> contained in those messages.

Sounds good.
Stephen

Re: #LayerPrimitives

[Stephen Colebourne]

On 17 March 2017 at 16:39, David M. Lloyd  wrote:
>> It is clear to me that JPMS does contain an API with some flexibility
>> here, but it is hard to judge whether my second expectation is met.
>> This is because the discussion is ledf by OSGI and JBoss Modules
>> migration concerns, rather than by consideration of what a new module
>> system would need. Perhaps these concerns are the same, perhaps not. I
>> would like to see a discussion from the EG about what blocks a
>> theoretical new extended module system built on top of JPMS.
>
> In the green field, all things are possible.
>
> You could create a variety of module systems that behave in a variety of
> ways and yet still adhere to the current JPMS restrictions.  If you're
> willing to discard, rewrite, or retool all existing software, and change
> best practices (even if the current practices are not particularly harmful
> or problematic, or even useful), and derive your acceptable use cases from
> the implementation restrictions (instead of the other way around), you can
> conform to anything and fit in any situation.

I think the evidence is that the JPMS approaches the problem space in
a different way to OSGI/JBossModules, so it is no surprise to find a
clash. These existing module systems still work in classpath mode. It
seems entirely reasonable that they don't work in module mode, since
they represent a clashing view of modularization. Thus, I'm suggesting
that it is right to consider the whole problem space again in terms of
JPMS, not just try to adapt existing projects.

>> it is more important to close out Java 9 and move on.
> Is it though?

Yes. The JPMS can be enhanced in future if necessary. The core team
has spent copious amounts of time on modules, and it is time we got
back to adding features that will benefit productivity, an area where
Java remains weak. Whether modules will be widely adopted remains open
to question, but modularizing the JDK itself will be a step forward,
even if that is the only benefit.

> Will they care that the community process was
> shunned in favor of expedience?

The JCP has tended to work OK for Java EE, but has never been a
suitable mechanism for driving SE. I view it as an artificial box
ticking process for SE - a zombie:
http://blog.joda.org/2011/06/java-se-7-passes-in-zombie-jcp_1590.html

IMO, the issues to resolve are module naming (where all the advice is
that short names will fail the wider community) and automatic modules
(where guessing names is not viable).

Stephen

Re: #LayerPrimitives

[Stephen Colebourne]

On 20 March 2017 at 10:47, Andrew Dinn  wrote:
> Regarding the first point, I have to ask why someone using an
> alternative module system to modularize their code would also want to
> use JPMS to modularize that same code.

The main feature that JPMS offers is the ability to encapsulate
package beyond what is possible today. That is useful behaviour beyond
the bounds of the default implementation of the JPMS. Thus, the intent
of an "alternative module system" I was describing was to use JPMS
modules with module-info in ways beyond that permitted by the default
module-path loader. Fortunately we do have Layers which do appear to
provide quite a lot of flexibility in this area.

> that /my/ perception of requests to support
> more 'dynamic' capabilities in JPMS is that they are based on a
> misguided desire to allow alternative module systems, specifically JBoss
> Modulss and OSGi, to operate hand in glove with JPMS rather than
> interoperate in a coherent and useful manner.

I broadly agree. Some Java 7 libraries work well with lambdas, others
were better with a major rewrite for java 8. The same applies to
existing frameworks and module systems - its OK to need a major
rewrite or redisgn for a major new version of the platform, something
that may lead to very a different approach to today.

Thanks for the additional detailed thoughts, which I found helpful.
Stephen

#LayerPrimitives

[Stephen Colebourne]

I note the discussions on various threads about JPMS primitives.

My first expectation is that OSGI and JBoss modules can run in Java 9
in classpath mode.

I believe this to be true today.


My second expectation is that there are sufficient primitive
operations within JPMS to allow a _new_ module system (not OSGI or
JBoss Modules) to be built that permits some but not all of the
features seen in OSGI and JBoss Modules. Thes should include:
- dynamic loading/reloading of modules
- lifecycle callbacks
- multiple versions of a class loaded at the same time
- handling of clashing packages
- some form handling for module cycles

It is clear to me that JPMS does contain an API with some flexibility
here, but it is hard to judge whether my second expectation is met.
This is because the discussion is ledf by OSGI and JBoss Modules
migration concerns, rather than by consideration of what a new module
system would need. Perhaps these concerns are the same, perhaps not. I
would like to see a discussion from the EG about what blocks a
theoretical new extended module system built on top of JPMS.

Specifically, I note Thomas Watson's concern over non-lazy class
loader construction as being a concern [1] (point 2) for module
systems.


My third expectation is that it should be possible for bytecode/class
generation tools to generate code inside a module, including in a
package that is not known by the module graph.

I note that Proxy appears to have a special case for "dynamic
modules", and am concerned that this may over-emphasise Proxy compared
to other similar tools that are not part of the JDK.


Note that while I understand the desire for OSGI and JBoss Modules to
fit into JPMS and get the benefits, I do not consider that necessary
to complete the JSR - it is more important to close out Java 9 and
move on. These existing systems work today, and provide benefits
today, something that won't change. That they won't get the enhanced
security benefits of JPMS modules is a secondary concern IMO.

Stephen

[1] 
http://mail.openjdk.java.net/pipermail/jpms-spec-observers/2017-March/000801.html

Re: #VersionsInModuleNames

[Stephen Colebourne]

On 15 March 2017 at 17:47, Alan Bateman  wrote:
> This is the consumer choosing a module name for a library that they don't
> maintain and renaming that library to match (you are writing the `requires
> X` before X exists). All I'm saying is that the library maintainer should be
> the one that chooses the module name. In its absence then deriving the name
> from from the library gives you a stable name in the short term.

I agree that the library maintainer should choose the module name, but
automatic modules will pretty much force consumers to do so, maing a
mess. The proposal above is for the case where person A has randomly
picked a modue name for a module that is not yet modularized, and
person B has to do something to comply. The proposal simply suggests
that the JPMS should not coonvert filename "bar-1.2.jar" to module
name "bar" - the developer should do this instead by renaming the
file.

Anyway, I'll write up more on how to avoid automatic modules in another thread.
Stephen

#VersionsInModuleNames

[Stephen Colebourne]

Responding to the discussion on the EG list.

I am now strongly of the opinion that using the language to restrict
module names to not end in a number is a mistake. The two clear cases
show two reasons why it is a mistake.

commons-lang3 is the third version of commons-lang. It was named  this
way because the API changed, and there was a strong desire to allow
both commons-lang and commons-lang3 to be on the classpath. Note that
the package for commons-lang is org.apache.commons.lang and the
package for commons-lang3 is org.apache.commons.lang3.

(Again, if we were to name modules after the highest package, it would
be obvious what the relationship is, and why restricting numbers
doesn't make sense, because they are allowed in package names)

fabric8 is a brand name, and an example of the future of naming
projects. Over time, more and more good project names are being used
up (just like good user names). Roll forward 10 years, and the chances
of finding a good project name will decrease. Using a number as part
of the name is one way to sidestep the problem and expand the number
of available names.

If the current #VersionsInModuleNames plan is not changed, the key
question is What Should These Projects Name Themselves?

commons-lang-three?
Maybe, but yuck.

fabricate?
Maybe, but really that is a completely different name, and the whole
point of using the 8 was to avoid using the -ate project name (maybe
someone else owns "fabricate").

I'm sure there are plenty of other examples on Maven Central. But it
doesn't really matter. Both of these are valid reasons to name a
project with a number at the end. As such, the #VersionsInModuleNames
proposal cannot stand.


Since part of the reason for this proposal was automatic modules,
which have caused problems in other ways, I'll suggest the following
change to automatic modules. Automatic modules must either contain the
Module-Name MANIFEST entry, or have a file name that exactly matches
the desired module name. ie. the standard jar files downloaded from
Maven Central, eg foo-bar-1.2 must be renamed to be used as an
automatic module. This means that the proposed rules for removing the
trailing version and converting dashes to dots would be removed from
the spec. Build systems and developers are perfactly capable of
renaming a file - baking specific naming conversion rules into the
JPMS is inadvisble.

(Note that the above is not an endorsement of automatic modules in
general, I believe that there are better solutions to the migration
problem. But the above is an imporvement to the current state of the
JPMS.)

thanks
Stephen

Re: How to name modules, automatic and otherwise

[Stephen Colebourne]

On 27 February 2017 at 20:44, Alan Bateman  wrote:
>> In summary, module naming must match package naming, because modules
>> are ultimately just a collection of packages with
>
> Some module don't contain any types/packages. Aggregator modules is one
> example (`java.se` and `java.se.ee`). Resource bundles (for
> i18n/translation) might only contain .properties files. Then there are
> modules that only contain native code, one example in the JDK
> is`jdk.jdwp.agent` (debugger agent).

While interesting, these are edge cases. A developer can always add
.xxx to a package name they control to get a name for the edge case.

Stephen

Re: How to name modules, automatic and otherwise

[Stephen Colebourne]

Having spent further time considering naming, I have come to the
conclusion there is only one acceptable solution - highest package
name (reverse DNS). It now seems to me that this can almost be
"proven" as follows:


Consider a module that contains three packages:

module ??? {
   exports com.foo.willow;
   exports com.foo.willow.model;
   exports com.foo.willow.util;
}

It is possible to refactor such a module into three separate modules,
where each module consists of one package:

module ??? {
   exports com.foo.willow;
   requires transient ???.model;
   requires transient ???.util;
}
module ??? {
   exports com.foo.willow.model;
}
module ??? {
   exports com.foo.willow.util;
}

Note here that as a general rule, it is always possible to refactor a
module into a number of single-package modules. (I can't actually
prove this offhand, but it seems an entirely reasonable claim).

Now, ask yourself the question - what should the module name be for
each single-package module?

There is only one possible, answer that is not completely ridiculous -
the module name must be the same as the package name. Any other answer
involves the invention of some other name that does not uniquely
reference the content of the single-package module.

module com.foo.willow {
   exports com.foo.willow;
   requires transient com.foo.willow.model;
   requires transient com.foo.willow.util;
}
module com.foo.willow.model {
   exports com.foo.willow.model;
}
module com.foo.willow.util {
   exports com.foo.willow.util;
}

Stepping back up to the original, it therefore follows that the module
name must be a summary of the packages included. Which means, in
general, the "highest" of the packages included/exported - aka reverse
DNS.

module com.foo.willow {
   exports com.foo.willow;
   exports com.foo.willow.model;
   exports com.foo.willow.util;
}

In fact, the primacy of package naming is fundamental to the whole
Jigsaw design. The JPMS won't load two modules where the packages
conflict. Exports and Requires are expressed at the package level too.
And the developer writing code is actually importing from a package,
not a module. There is thus no getting away from the fact that modules
are built on top of the existing package concept, simply providing
packages with greater security.

Given all this, a module name SHOULD match one of the packages it contains.

In essence, there are two worlds that JPMS seeks to keep separate -
development and build system. The build system world has versions,
artifacts, groups, organizations, jar-files, project names, etc - none
of which affect bytecode. The development world has class, package and
module names - all ending up in bytecode. Module names should
therefore be completely uninfluenced by versions, artifacts, groups,
organizations, jar-file names and project names. The fact that
jar-files are being used to package modules is a big confusion to this
whole naming debate, as the jar-file name is from the build system
world, not the development world.

With regards to automatic modules, this implies that automatic modules
must contain the Module-Name MANIFEST information (as it takes too
long to scan a jar for packages). Allowing the command line to map jar
files to module names would also be acceptable.


On 16 February 2017 at 16:48,  <mark.reinh...@oracle.com> wrote:
> I do know, however, of at least
> one major, well-known project whose developers intend to adopt the
> project-name-prefix convention for their module names.

Mark indicates that some projects are intent on using short names. To
stop this (where developers try and be "clever" and anti-social) the
module-info compiler and/or jar tool MUST emit a warning (I'd prefer
error, but there are probably some edge cases). The warning would be
something like "the module name must match or be related to one of the
contained packages". While this does not absolutely prevent developers
doing the wrong thing, it would provide additional force to the rule.
Potentially Maven Central could also validate this.


In summary, module naming must match package naming, because modules
are ultimately just a collection of packages with some additional
security rules. Any module can always be reduced to a number of
single-package modules, and since there is only one possible name a
single-package module should have, the combined module should have a
name based on the packages contained - which will almost certainly be
the highest contained package name.

Stephen



On 16 February 2017 at 23:19, Stephen Colebourne <scolebou...@joda.org> wrote:
> On 16 February 2017 at 16:48,  <mark.reinh...@oracle.com> wrote:
>>  This can be done very simply, with a single new JAR-file manifest 
>> `Module-Name` attribute
>
> I welcome this.
>
>> The reversed domain-name approach was sensible in the early days of Java,
>> before we had development tools sophisti

Re: How to name modules, automatic and otherwise

[Stephen Colebourne]

On 17 February 2017 at 00:15, Stephan Herrmann
 wrote:
>> module mainlib from com.mycompany {
>>   requires base;  // implicit, favours group 'com.mycompany' if there is a
>> clash
>>   requires willow;  // uses 'com.mycompany' because there is a clash
>>   requires willow from org.joda;  // explicitly specified, but only
>> needed to resolve a clash
>> }
>
>
> From here, wouldn't it be trivial to change Mark's counter example:
>
> module com.bar:foo.data {
> exports com.bar.foo.data;
> requires org.hibernate:hibernate.core;
> requires org.hibernate:hibernate.jcache;
> requires org.hibernate:hibernate.validator;
> }
>
> into a positive example:
>
> import org.hibernate:*;
> module com.bar:foo.data {
> exports com.bar.foo.data;
> requires hibernate.core;
> requires hibernate.jcache;
> requires hibernate.validator;
> }

There is a subtle difference through. With the two layouts above, the
groupId is always specified. With what I suggested, the groupId is
late binding in the common case, giving a degree of flexibility to
find the matching module.

module com.bar:foo.data {
exports com.bar.foo.data;
requires hibernate.core;
requires hibernate.jcache;
requires hibernate.validator;
}

ie. this looks for hibernate.core on the module path and might find it
from group org.hibernate or from another group. Finding it twice =
error. This flexibility may be good or bad depending on how you view
the world.

BTW, one advantage of groups over reverse DNS is that it forces a
consistent naming scheme on everyone, whereas the current proposal
could well result in a mismash of styles.

Stephen

Re: How to name modules, automatic and otherwise

[Stephen Colebourne]

On 16 February 2017 at 16:48,   wrote:
>  This can be done very simply, with a single new JAR-file manifest 
> `Module-Name` attribute

I welcome this.

> The reversed domain-name approach was sensible in the early days of Java,
> before we had development tools sophisticated enough to help us deal with
> the occasional conflict.  We have such tools now, so going forward the
> superior readability of short module and package names that start with
> project or product names is preferable to the onerous verbosity of those
> that start with reversed domain names.

What tools?

With short identifiers clashes are inevitable. Because the module name
is baked inside the module in binary format, the only way to resolve
the clash is to rewrite the module. The Java platform has not demanded
anything like this before, and I can't see how it meets the reliable
configuration requirement. Rewriting modules as part of the build
system is a red line for me. I need to be able to see that the module
on the module path is the same bits as that from the source of jars.

The standard case to consider is as follows:

- In 2017, a company creates an internal foundation library called
"willow" and it becomes very popular within the company and is used
100s of times
- In 2018, an unrelated open source project starts up with the name
"willow" and becomes very popular. Both now publish modules with the
name "willow" (one privately, one publicly).
- In 2019, the company wants to use the open source "willow" library
(directly or indirectly), but can't due to name clash
- In 2020, the company wants to open source their "willow" library,
but can't due to name clash

The plan outlined, favouring short IDs, provides no solution to this
problem that I can see. There simply isn't the breadth of identifier
to avoid clashes like this (you can't possibly predict the future
where you might need to coexist with an open source module that
doesn't even exist yet). Proposal (A) only tackles automatic modules,
and not the bigger problem where names are baked into the module
itself.

The simplest and most consistent option is reverse DNS everywhere.
Everyone understand it and few will object!

An alternative option would be that open source can use short names,
but companies "must" use reverse DNS. But this is far from ideal given
how projects move from private to public, or how companies merge.

Another alternative is some form of group, that may or may not map
onto maven's group, where most of the time it does not have to be
specified:

module mainlib from com.mycompany {
  requires base;  // implicit, favours group 'com.mycompany' if there is a clash
  requires willow;  // uses 'com.mycompany' because there is a clash
  requires willow from org.joda;  // explicitly specified, but only
needed to resolve a clash
}

With this approach, the clash can be resolved, but only needs to be by
the first module in the graph to pull both in. Any transitive use of
the two willow modules would be fine.

In summary, I recognise the desire for short, pretty identifiers.
However, I remain of the opinion that they are highly dangerous for
the wider ecosystem without some additional ability to qualify them.
The are many more private jars than public jars, and the clashes seen
today on Maven Central are just the tip of the iceberg of this
problem.

Stephen

Re: Automatic module names - "requires package" proposal

[Stephen Colebourne]

On 3 February 2017 at 15:40, Alan Bateman  wrote:
> I think automatic modules are critical to migration, otherwise things move
> at the pace of the slowest project (which might be 0km/h in case of
> unmaintained projects). A lot of the discussion here has been on the naming
> but automatic modules but the other important benefit of automatic modules
> is that they support bridging to the class path.
>
> As regards the example naming clash then these two projects might already
> get complaints over their poor choice of artifacts, esp. when artifacts for
> both projects are in same directory (say where someone distributes with all
> JAR files in a `lib` directory).

We can't close our eyes to this problem. There can be naming clashes
as above, naming problems due to having to guess a project's choice of
a module name. We need to take the guesswork out, and have something
based on Java compilation elements.

As such, I'd like to propose an alternative based on packages that
could still be feasible in the time available:

1) Have two types of `requires` clause - one for modules and one for packages
2) When requiring a package, that package must be available, either on
the classpath or in a module
3) When requiring a package, the package must not be found in a module
that is explicitly referenced
4) Fully modularized applications do not have `requires package` -
this is a migration feature like `open modules`

module com.org.a {
  // dependency on a module (modular jar)
  requires module com.org.b;
  requires module com.org.c;

  // dependency on packages (modular jar or classpath)
  requires package com.org.d;
  requires package com.org.d.x;
  requires package com.org.d.y;
}

In the example above, the three explicitly specified packages must not
be found in module `com.org.b` or `com.org.c` (rule #3). They must
instead be found in a jar on the classpath (each package must be found
only in one jar), or in a future module (which could have any name,
not just `com.org.d` or `d`).

The `requires package` provides the "bridge" to the classpath and
older jars that is desired, without the mess of automatic modules.
Note that there would be no automatic modules as currently planned in
this proposal.

(FWIW, in the implementation, I'd imagine that each required package
would be created as a separate module, using the same openess/security
design as automatic modules today)

Stephen

Automatic module names

[Stephen Colebourne]

Back in October, I raised the issue of modules names generally and for
automatic modules specifically [1]. The short thread came to no
conclusion, but recent threads have again raised similar problems. The
problem is that automatic modules have magical name creation from a
filename, which is brittle and unlike anything else in Java.

I also recently looked at the Joda-Convert and Joda-Beans libraries,
to see if I could add module-info in preparation for Java 9. I quickly
backed away, again because of the same issue. Put simply, I am
unwilling to write a module-info file that refers to a dependency that
is not yet a module. And I have to advise all open source projects to
do the same. Given this, there can be no simple migration to the JPMS
for open source projects. Each open source project must wait for all
its dependencies to migrate to JPMS (by adding a module-info and
publishing to Maven Central).

The issue is clear. If I write this:

module org.joda.convert {
  requires guava;
}

where guava is an automatic module, I am locking in the name of the
guava dependency, something that I do not control. The name "guava" is
just a guess. The guava authors might choose "com.google.guava" or
something else entirely.

In a closed system of modules, ie. a private application, automatic
modules are fine, because the requires clause can be changed if it
turns out the guess was wrong. But once published as an open source
project to Maven Central or elsewhere, the guess cannot be fixed if it
is wrong (without releasing a new version of the library, which is not
an acceptable solution).

I also strongly believe that module names cannot be flat and
unstructured, such as "joda-convert" or "guava". They must have
structure, such as the domain name or a Maven-style group name
"org.joda.convert" or "org.joda:joda-convert". The potential for
clashes has been shown by the Maven team [2].

Some brainstormed possible changes:

- Remove the automatic module concept altogether

- Define a clear mapping from Maven Central co-ordinates to module
name that includes the group, artifact and classifier

- Provide a text file to JPMS that allows incorrect module names to be
mapped to the correct name

- Publicly advise against using automatic modules for open source projects

- Change rules of Maven Central to prevent modular jars being added
that depend on an automatic module

- Allow requires clauses to have aliases - requires org.guava.guava OR guava.

- Allow modules to have aliases - module org.guava.guava AKA guava


Given that applications can depend on libraries that haven't been
released in years, this has the potential to be a critical problem for
the ecosystem. My preference remains to define a clear mapping from
the widely adopted Maven Central naming strategy to JPMS modules.
Ideally, this would be a formal group concept in the JPMS, something
that I believe is sorely lacking.

Stephen

[1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-October/009631.html
[2] 
http://mail.openjdk.java.net/pipermail/jpms-spec-observers/2017-January/000707.html

Re: #CompileTimeDependencies and module resolution

[Stephen Colebourne]

The standard use case for the feature is for libraries with optional
dependencies:

module Lib { requires static A; requires B; }
module A { ... }
module B { ... }

Given this setup:
 module App1 { requires Lib; }
the module graph should include App1, Lib and B.
Any use of A from Lib must be guarded, as A is not present.

Given this setup:
 module App2 { requires Lib; requires A }
the module graph should include App1, Lib, A and B.
Module A will be visible and read by Lib.

ie. the optional depenedency expresses the concept of "use module A if
it is available, otherwise ignore it"


The --add-modules flag is only relevant when using the command line to
turn setup #1 into setup #2, something which should be rare.

Stephen


On 13 January 2017 at 11:33, Alan Bateman  wrote:
> On 13/01/2017 11:08, Remi Forax wrote:
>
>> Hi Sander,
>> you're right, it's a bug, --add-modules should not be necessary.
>>
>> Rémi
>
> I don't think there is a bug here. Instead the example that Sander has
> chosen doesn't resolve a module that `requires B`. The "Notes" section in
> #CompileTimeDependences proposal has the rational for this. If the example
> is extended to:
>
> module A { requires static B; requires C; }
> module B { ... }
> module C { requires B; }
>
> then the resulting module graph will have contain at least A, B and C, and A
> will read at least B and C.
>
> -Alan

Re: MethodHandle performance

[Stephen Colebourne]

Thanks Claes and everyone else who took a look. The asSpreader() trick
does indeed work.

So it looks like method handles can be a viable alternative to
reflection for Jigsaw, but users of method handles do need to take
care about the pattern of usage of method handles to get performance.
Hopefully JDK-8078511 will get addressed at some point too.

(Part of the problem I suspect is that because fewer people have
looked at method handles, there is less experience of how to use them
effectively on the web)

thanks
Stephen


On 12 January 2017 at 20:29, Claes Redestad <claes.redes...@oracle.com> wrote:
> Right, I was just looking at the micro Stephen provided me, and it does
> seem that the added cost for this case is due to invokeWithArguments
> creating a new invoker every time.
>
> This appears to be a known issue[1], so a design that can use invoke or
> invokeExact instead should be considered:
>
> -this.constructor = constructorHandle;
> +this.constructor = constructorHandle.asSpreader(Object[].class,
> propertyTypes.size());
>
> ...
>
> -return (T) constructor.invokeWithArguments(args);
> +return (T) constructor.invoke(args);
>
> This seems sufficient to get the MH-based implementation on par with
> the reflection based one in all tests.
>
> Thanks!
>
> /Claes
>
> [1] https://bugs.openjdk.java.net/browse/JDK-8078511
>
>
> On 2017-01-12 20:46, Michael Rasmussen wrote:
>>
>> Doing a quick JMH benchmark here, with a constructor taking 4
>> arguments, invokeWithArguments was about 15-30x slower than invoke and
>> invokeExact.
>> Otherwise, static final MH was 2.5x faster then reflection, and
>> instance field MH was 1.3x faster
>>
>> For output and test code:
>> https://gist.github.com/anonymous/32e698d04c7456c10f069686072d7cc6
>>
>> /Michael
>>
>> On 12 January 2017 at 20:12, Stephen Colebourne <scolebou...@joda.org>
>> wrote:
>>>
>>> @Claes
>>>
>>> Fat fingers - should have been 1.8.0_112
>>>
>>> The code is in the link:
>>>
>>> https://github.com/JodaOrg/joda-beans/commit/ad7d61a7ff72f932957127117dd8f377e1e2bf60
>>>
>>> where a HandleMetaBean (method handle) performs the same job as
>>> LightMetaBean (reflection) as shown in the non-JMH test
>>> TestHandle.main().
>>>
>>> @Jochen
>>>
>>> The method handle setup code is all done in static initializers. The
>>> performance tests are all testing runtime usage, excluding the static
>>> initializer setup cost (as one off startup costs are not relevant to
>>> my use case).
>>>
>>> @Remi
>>> Thanks for taking a look. Its the constructor invocation that is most
>>> obviously a problem. In HandleMetaBean.build(Object[]) where it uses
>>> invokeWithArguments(Object[]). I get 1650ms for reflection vs 3500ms
>>> for method handles.
>>>
>>> The meta-property getter code runs much quicker, so a JMH test would
>>> really be needed to confirm the difference there.
>>>
>>> The asType() and invokeExact() code was added to see if that made a
>>> difference, but it did not.
>>>
>>> @Aleksey
>>> I saw you post when researching before writing my mail today. The code
>>> cannot use static final method handles (I doubt there are many use
>>> cases for those, to be honest). The goal of the code here is to obtain
>>> an object implementing my interfaces that can be invoked in a general
>>> way to invoke a constructor or a getter. As such, the method handles
>>> need to be instance variables.
>>>
>>> I have now done a JMH test.
>>>
>>> The good news is that the method handle for the getter is slightly
>>> faster when taken in isolation:
>>>
>>> JodaBenchmark.testMethodHandleGetavgt   508.421 ± 0.078  ns/op
>>> JodaBenchmark.testReflectionGet  avgt   50   11.003 ± 0.050  ns/op
>>>
>>> The bad news is that the method handle constructor call is not 2x
>>> reflection, but 6x:
>>>
>>> JodaBenchmark.testMethodHandleBuild  avgt   50  219.212 ± 2.400  ns/op
>>> JodaBenchmark.testReflectionBuildavgt   50   36.012 ± 0.167  ns/op
>>>
>>> This test reduced the difference to :
>>>return (T) constructorHandle.invokeWithArguments(args);
>>> vs
>>> return constructor.newInstance(args);
>>>
>>> Email me privately for a zip of the JMH test:
>>>
>>> Any thoughts on the 6x slower call? thanks for looking
>>> Stephen
&

Re: MethodHandle performance

[Stephen Colebourne]

@Claes

Fat fingers - should have been 1.8.0_112

The code is in the link:
https://github.com/JodaOrg/joda-beans/commit/ad7d61a7ff72f932957127117dd8f377e1e2bf60

where a HandleMetaBean (method handle) performs the same job as
LightMetaBean (reflection) as shown in the non-JMH test
TestHandle.main().

@Jochen

The method handle setup code is all done in static initializers. The
performance tests are all testing runtime usage, excluding the static
initializer setup cost (as one off startup costs are not relevant to
my use case).

@Remi
Thanks for taking a look. Its the constructor invocation that is most
obviously a problem. In HandleMetaBean.build(Object[]) where it uses
invokeWithArguments(Object[]). I get 1650ms for reflection vs 3500ms
for method handles.

The meta-property getter code runs much quicker, so a JMH test would
really be needed to confirm the difference there.

The asType() and invokeExact() code was added to see if that made a
difference, but it did not.

@Aleksey
I saw you post when researching before writing my mail today. The code
cannot use static final method handles (I doubt there are many use
cases for those, to be honest). The goal of the code here is to obtain
an object implementing my interfaces that can be invoked in a general
way to invoke a constructor or a getter. As such, the method handles
need to be instance variables.

I have now done a JMH test.

The good news is that the method handle for the getter is slightly
faster when taken in isolation:

JodaBenchmark.testMethodHandleGetavgt   508.421 ± 0.078  ns/op
JodaBenchmark.testReflectionGet  avgt   50   11.003 ± 0.050  ns/op

The bad news is that the method handle constructor call is not 2x
reflection, but 6x:

JodaBenchmark.testMethodHandleBuild  avgt   50  219.212 ± 2.400  ns/op
JodaBenchmark.testReflectionBuildavgt   50   36.012 ± 0.167  ns/op

This test reduced the difference to :
   return (T) constructorHandle.invokeWithArguments(args);
vs
return constructor.newInstance(args);

Email me privately for a zip of the JMH test:

Any thoughts on the 6x slower call? thanks for looking
Stephen


On 12 January 2017 at 14:47, Claes Redestad <claes.redes...@oracle.com> wrote:
> Hi Stephen,
>
> this is surprising; handles should typically be as fast or much
> faster than reflection (VarHandles can be faster than Unsafe in certain
> cases), but may carry a slightly more expensive setup cost - do you have a
> reproducer I could try?
>
> 8b122 - do you mean 8u122 EA?
>
> Thanks!
>
> /Claes
>
>
> On 2017-01-12 15:23, Stephen Colebourne wrote:
>>
>> I've recently tried [1] converting Joda-Beans use of setAccessible()
>> to use MethodHandle. Since it is a code generator, the actual coding
>> is relatively easy, and obtaining the MethodHandles.Lookup instance
>> with the "private" capability is simple. While the MethodHandles API
>> looks very complex, it isn't too bad to use, although it is
>> undoubtedly more complex than reflection.
>>
>> (Note that the standard Joda-Beans technique is to code generate
>> normal Java code to avoid the need to use reflection, but it can
>> optionally generate reflection-based code in "light bean" mode. It is
>> that reflection approach that is being examined here).
>>
>> The real problem however is performance. In my tests, I am seeing a
>> MethodHandle approach being 2 to 3 times slower than a reflection
>> approach for identical functionality, which is quite a significant
>> degradation. (using Java 8 b122)
>>
>> Given the performance, I left to question whether the repeated Jigsaw
>> advice to use MethodHandle instead of setAccessible is viable - in the
>> kinds of places that use reflection, performance tends to be critical.
>> Is there, or has there been, work in Java 9 to improve the performance
>> of method handles?
>>
>> Stephen
>>
>> [1] https://github.com/JodaOrg/joda-beans/commits/wip/methodhandles
>>
>

MethodHandle performance

[Stephen Colebourne]

I've recently tried [1] converting Joda-Beans use of setAccessible()
to use MethodHandle. Since it is a code generator, the actual coding
is relatively easy, and obtaining the MethodHandles.Lookup instance
with the "private" capability is simple. While the MethodHandles API
looks very complex, it isn't too bad to use, although it is
undoubtedly more complex than reflection.

(Note that the standard Joda-Beans technique is to code generate
normal Java code to avoid the need to use reflection, but it can
optionally generate reflection-based code in "light bean" mode. It is
that reflection approach that is being examined here).

The real problem however is performance. In my tests, I am seeing a
MethodHandle approach being 2 to 3 times slower than a reflection
approach for identical functionality, which is quite a significant
degradation. (using Java 8 b122)

Given the performance, I left to question whether the repeated Jigsaw
advice to use MethodHandle instead of setAccessible is viable - in the
kinds of places that use reflection, performance tends to be critical.
Is there, or has there been, work in Java 9 to improve the performance
of method handles?

Stephen

[1] https://github.com/JodaOrg/joda-beans/commits/wip/methodhandles

Re: Proposal: #VersionedDependences

[Stephen Colebourne]

On 9 December 2016 at 21:46,   wrote:
> When compiling a module that depends on some other modules, record the
> version strings of those modules, if available, in the resulting module
> descriptor.

Overall this seems like a good idea, for diagnostic reasons.

> Now that compile-time versions can be recorded in module descriptors
> there is even less need to tolerate version information in module names,
> a bad practice that we'd like to discourage at the outset.  We therefore
> further propose to:
>
>   - Revise the accepted proposal for #VersionsInModuleNames [3] to state
> that a module name appearing anywhere in a source-form module
> declaration must both start and end with "Java letters" [4].

I continue to think that this is unwise. There are legitimate reasons
for projects to want to have a number at the end of the project name -
fabric8 and lang3 being mentioned already.

Stephen

Re: Proposal: #ReflectiveAccessToNonExportedTypes (revised) & #AwkwardStrongEncapsulation: Weak modules & private exports

[Stephen Colebourne]

On 11 October 2016 at 16:14,   wrote:
> A couple of other problems I see with Colebourne's approach:
>
>   - The only way to allow an exported API package to be used reflectively
> is to expose it for deep reflection.  That requires typing another
> directive and, worse, makes the internals of that package available
> for deep reflection, which is probably not what was intended.

Having re-read my proposal, you are correct, however it wasn't what I
intended. My intention was that 'exports' included reflection on
public types (shallow reflection) and 'exposes' was necessary for
access to non-public types (deep reflection).

>   - The interactions between `exports`, `exposes`, and the qualified
> forms of these directives turns out to be pretty complicated once you
> work out all the cases.  (We've done this internally, for a somewhat
> similar proposal that allows both `private` and `dynamic` modifiers
> on the `exports` directive.)  They wind up being nearly as bad as the
> existing rules for protected members [2].  I'm not convinced that
> this relatively small bit of expressive power is worth the additional
> complexity.

I suspect that this complexity comes from interaction with the "to"
clause of exports/exposes. (4 basic variations - not exported,
standard export, exposed not exported, exposed & exported, plus the
various combinations of where things are exported/exposed to)

However, I also consider the compromise of the Reinhold proposal, that
there is no way to export for deep reflection without also exporting
for normal IDE use, is simply not acceptable. Too many modules will
need deep reflection, that the current proposal essentially means that
the strong encapsulation requirement cannot be met.

It is worth noting that the debate since the latest proposal came out
has primarily been about deep reflection and how to allow that in the
system There has been little disagreement that the "standard" type of
strong encapsulation should only include shallow reflection on public
types.

Taking a step back, the following seems like simplest thing that would work:

weak module foo {
  requires...
  // all packages exported
  // all packages accessible for deep reflection
}

exposed module foo {
  requires...
  exports...
  // all packages accessible for deep reflection
}

module foo {
  requires...
  exports...
  // strongly encapsulated, no deep reflection
}

(Happy to see alternative keywords proposed instead of weak/exposed)

This provides a simple sliding scale of encapsulation - weak, exposed
for deep reflection and strong. And there should be no combinatorial
explosion of complexity.

What is lost is the ability to only expose some packages for deep
reflection, or to expose deep reflection only to certain modules. But
I am very comfortable with that compromise - it seems like once you
grant deep reflection, you are giving the keys away to the whole
module, and trying to restrict it further than the module-level is a
game of diminishing returns.

Consider this Colebourne proposal #2 ;-)

Stephen

Re: Module names - related to #ModuleNameSyntax

[Stephen Colebourne]

On 10 October 2016 at 13:47, Andrew Dinn <ad...@redhat.com> wrote:
> On 10/10/16 12:47, Stephen Colebourne wrote:
>> At some later time, guava is modularised, but chooses the sensible
>> module name "com.google.guava". Now, the module foo (released as open
>> source to maven central) has the wrong module name for guava.
>
> I think this is where your argument does it's sleight of hand trick
> (intentional or not) that makes people believe in magic and ask for
> protection from it.
>
> Let's just rename your modular jar to foo-1.0.jar and see how the
> sleight of hand happens.
>
> You have a released module foo-1.0.jar that depends on and works with
> non-modular jar guava-19.0.jar. Now modular jar guava-20.0.jar is
> released. So, to cope with the update to guava you do what is normally
> done to cope with a dependency upgrade i.e. you release modular jar
> foo-1.1.jar which you have rebuilt to require com.google.guava.
>
> At this point foo-1.0.jar is not broken, nor is foo-1.1.jar. They work
> with the appropriate versions of guava.jar and any user of foo has to
> respect those dependencies (just like any other released artifact).
>
> Where is the actual problem?

The concept of upgrading a version of a dependency doesn't apply to
the JPMS as it does not have versions. ie. foo-1.0.jar does not depend
on guava-19.0, it depends on guava (version undefined). There should
be no reason to release foo-1.1 - the JPMS design expects later
versions to be able to be substituted on the module path (provided
they are compatible, which JPMS doesn't really check).

But more significantly try multiplying the problem described by a tree
of dependencies, where each is upgraded on different cycles. Releasing
your own new version of the top library is simply not a practical
solution to the problem.

I certainly think that automatic modules are a form of magic in the
JPMS, and we might be better off without them. But this thread is
about the more general problem of naming modules in a global world,
something where reverse domain naming has served Java extremely well.

Stephen

Module names - related to #ModuleNameSyntax

[Stephen Colebourne]

"At JavaOne I asked a question about the naming standards for modules
given that we have automatic modules.

The scenario is as follows:
- an open source module foo that depends on Google guava
- guava has not yet been modularised
- guava is provided by jar file guava-19.0.jar and used as an automatic module
- foo declares "requires guava", where "guava" is a module name
derived from the jar file name

At some later time, guava is modularised, but chooses the sensible
module name "com.google.guava". Now, the module foo (released as open
source to maven central) has the wrong module name for guava.

Given this scenario, the Jigsaw team suggested that the correct module
name for guava is "guava". (The only alternative strategy at the
moment would be for open source projects published to maven central to
*never* declare requires on an automatic module).

I, and others, have expressed concern about a naming strategy for
modules that is not based on the reverse domain name strategy, eg
com.google or org.joda. It seems common to me that companies will have
modules that have the same name as publicly available modules in maven
central. The migration problem described above also seems overly
restrictive.

In addition, there are related problems to do with projects that fork
but want to retain the same name (maybe not desirable, but GutHub
forking is common).

Proposal
-
- a new concept *group name* should be introduced above module name
- the combination groupName:moduleName should always match the maven
group:artifact coordinates
- messaging in this way to the community will avoid migration issues
as the standard is already in use and accepted
- in module-info.java, the user must always specify both the group and
module name
- the requires clause may specify or omit the group name but always
has the module name
- best practice would be to include the group name to ensure reliable
configuration
- when depending on automatic modules, the group name would be omitted
- if omitted, the group name is inferred by the system from the
available modules on the module path
- automatic modules are in the unamed group

With this setup, the migration problem outlined above disappears. The
fully qualified name of guava would be "com.google.guava:guava", as
per maven naming standards [1]. Anybody who had depended on the
automatic module for guava using "requires guava" would still work (in
all except edge cases).

This would look for a module named "guava" in any group (likely to be
an automatic module). If found in two groups, the system fails to
start:
module com.mycompany:foo-utils {
  requires guava;
}

This would look for a module named "guava" in group "com.google.guava:
module com.mycompany:foo-utils {
  requires com.google.guava:guava;
}


This relates to #ModuleNameSyntax, in that the module name will be
more distinct - either with a colon, or being the short form. Ideally,
the convention would be for the module name to use dashes, not dots.
Thus, Joda-Beans would be "org.joda:joda-beans" when fully qualified.

Given the widespread adoption of maven, the combination of
group:artifact is very well known and accepted. It makes sense for
Java to adopt the convention.

(In fact, the way this is defined, it should be possible for Maven
Central / JCenter to run an automated job that adds a "weak module"
definition to all existing jars, using the maven coordinates as the
name. Whether this desirable to actually is debatable, but the fact
that it could be done strikes me as a good sign for migration.)

Stephen

[1] 
http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22com.google.guava%22%20AND%20a%3A%22guava%22

Re: Class names in java.lang.Module

[Stephen Colebourne]

Most coding only uses the simple name, not the fully qualified one,
and Configuration does occur in other projects [1].

The original poster referred to the package, where Configuration is
the only non-exception class that does not have "Module" in the name
[2].

Stephen

[1] 
https://commons.apache.org/proper/commons-configuration/apidocs/org/apache/commons/configuration2/Configuration.html
[2] 
http://download.java.net/java/jdk9/docs/api/java/lang/module/package-summary.html


On 26 September 2016 at 10:25, Neil Bartlett <njbartl...@gmail.com> wrote:
> Module is already in the name: “java.lang.module.Configuration”. Wouldn’t 
> “java.lang.module.ModuleConfiguration” look really odd?
>
> Neil
>
>> On 21 Sep 2016, at 16:18, Stephen Colebourne <scolebou...@joda.org> wrote:
>>
>> I had the same thought while watching the slides. Configuration is
>> certainly a class name that exists other places, and would benefit
>> from being ModuleConfiguration. Layer is less common, so not worried
>> so much. Exceptions with "Module" in the name like
>> ModuleNotFoundException would also be clearer.
>> Stephen
>>
>> On 21 September 2016 at 03:36, Kasper Nielsen <kaspe...@gmail.com> wrote:
>>> Hi,
>>>
>>> I was wondering if there are any reasons for why these 3 classes in
>>> java.lang.Module
>>>
>>> Configuration
>>> FindException
>>> ResolutionException
>>>
>>> Does not include the name Module?
>>> I especially am not to fond of the very generic Configuration name in my
>>> source code would much prefer something like ModuleConfiguration.
>>>
>>> Best
>>>  Kasper
>

Re: Alternative mechanism for reflective access control (#ReflectiveAccessToNonExportedTypes / #AwkwardStrongEncapsulation)

[Stephen Colebourne]

Having read this proposal a number of times, and considering how the
talks explained things at JavaOne, I have come to the conclusion that
this proposal is too complex. FWIW, I like the idea that a module
should be able to declare that it needs reflective access from its
users, however given that the proposal is what results from the idea,
it doesn't seem as appealing as it should.

The reason why I put forward the exports/exposes approach [1] is that
it keeps the questions that must be asked when creating a module
simple:
- what do I depend on publicly (requires)
- what do I publish publicly (exports)
- what do I publish privately (exposes)

>From a security point of view it also seems that it should be the
responsibility of a module to allow the publishing of its private
details, and simply depending on another module seems very minimal
(and easy to miss) as a mechanism to allow that extra permission.

Stephen

[1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-September/009370.html


On 21 September 2016 at 17:39, David M. Lloyd  wrote:
> In our internal discussion of the proposal for
> #ReflectiveAccessToNonExportedTypes, we discussed the ins and outs of
> various behaviors and have come up with a few ideas or starting points for
> solutions that we think would be more workable in conjunction with existing
> middleware (ours and others').
>
> For reasons previously explained, we do not think that weak modules are a
> good way forward; I won't go into that again here.  But the logical
> re-starting point is: If not weak modules, then what?
>
> I will boil it down to a few basic requirements that we have established.
> This list is probably non-exhaustive but hopefully complete enough to go on
> for now:
>
> • A module definition must be able to establish that a dependent has (or all
> modules have) access to one or more (or all) packages for public reflection
> only.
> • A module definition must be able to establish that a dependent has (or all
> modules have) access to one or more (or all) packages for public or private
> reflection only.
> • A module definition must be able to establish that a dependent has (or all
> modules have) access to one or more (or all) packages for public reflection
> and compilation/linkage (i.e. it's an export by today's terminology).
> • A module definition must be able to establish that a dependent has (or all
> modules have) access to one or more (or all) packages for public or private
> reflection and compilation/linkage (i.e. it's a "private" export by today's
> terminology).
> • As today, any packages not declared in one or more of the above categories
> is inaccessible outside of the module in any way (note that as I showed
> previously we have also concluded that it should continue to be impossible
> to export a package for compilation/linkage without public reflection, as we
> have not discovered any use for such a mode).
>
> More generally:
>
> • The syntax for all of the above has no particular constraint (in fact I
> will try to actively avoid touching what could be a very bikeshedding-rich
> discussion), except that it should not be construable as being pejorative
> against the usage of reflective frameworks; rather, it should be clear what
> level of trust is being established without raising undue warning.
> • Applications should not need gratuitous amounts of declarations in their
> module(s) in order to utilize frameworks.
> • As previously established, it should not be possible for one declaration
> to reduce the scope of access of another declaration in a module definition.
> • Access to a module (for reflective purposes only) must not cause conflicts
> if multiple such modules which contain identical packages are accessible to
> a single consumer; in other words, reflection-only access into
> non-dependency modules is not bound by duplicate package restrictions as
> long as each package is unique per class loader, as per the current (Java 8)
> class loader rules.
>
> The above cover the useful access modes that we have identified.  This is
> _nearly_ adequate to cover the use cases that we are currently concerned
> about; for example, I could export all packages for public reflection only
> to a specific framework, if only I know the module name of the
> implementation.
>
> Unfortunately, this does not work well in the case where a module may
> consume a framework whose specification is separate from the implementation.
> An application module may need to use (say) EJB and JPA; there is presently
> no clean way to do so without either (a) relying on a container environment
> to rewrite the descriptor or (b) opening up the module and defeating the
> security mechanism (e.g. "weak").  Without either of these workarounds, the
> application developer must have a good deal of knowledge about what modules
> provide what services within a framework-rich environment, possibly
> resulting in a very verbose (and error-prone) 

Re: Class names in java.lang.Module

[Stephen Colebourne]

I had the same thought while watching the slides. Configuration is
certainly a class name that exists other places, and would benefit
from being ModuleConfiguration. Layer is less common, so not worried
so much. Exceptions with "Module" in the name like
ModuleNotFoundException would also be clearer.
Stephen

On 21 September 2016 at 03:36, Kasper Nielsen  wrote:
> Hi,
>
> I was wondering if there are any reasons for why these 3 classes in
> java.lang.Module
>
> Configuration
> FindException
> ResolutionException
>
> Does not include the name Module?
> I especially am not to fond of the very generic Configuration name in my
> source code would much prefer something like ModuleConfiguration.
>
> Best
>   Kasper

Re: Java 9 EA 136 error

[Stephen Colebourne]

On 20 September 2016 at 07:49, Alan Bateman  wrote:
> What does `java -version` show? I ask because "weak module myapp { }" should
> compile with the EA builds.

Sorry I wasn't clear. The "weak module" part compiles now.

Stephen

Re: Java 9 EA 136 error

[Stephen Colebourne]

Just to note that the report below is mostly about the confusing error
message, something which users will get if they write "wek" instead of
"weak". (I've downloaded the correct build now and it has the same
error message issue)

thanks
Stephen


On 20 September 2016 at 05:58, Alan Bateman <alan.bate...@oracle.com> wrote:
>
> On 19/09/2016 22:25, Stephen Colebourne wrote:
>>
>> When trying to compile a module-info.java file, "weak" is rejected (I
>> assume not implemented yet).
>
> Can you the Jigsaw EA download [1]? The regular JDK 9 downloads don't have
> support for the design issues that are till under discussion.
>
> -Alan
>
> [1] https://jdk9.java.net/jigsaw/

Java 9 EA 136 error

[Stephen Colebourne]

When trying to compile a module-info.java file, "weak" is rejected (I
assume not implemented yet). But, this is a report about the error
handling, which fails to recognise that this is a module-info file,
and thus outputs messages about missing class/interface/enum. Some
additional processing is needed to work out that the user was trying
to declare a module and got the syntax wrong:

>\apps\jdk-9-ea\bin\javac --module-source-path src -d mods src\myapp\
org\myapp\MyApp.java src\mylib\org\mylib\MyLib.java src\myapp\module-info.java s
rc\mylib\module-info.java

src\myapp\module-info.java:1: error: class, interface, or enum expected
weak module myapp {
^
src\myapp\module-info.java:3: error: class, interface, or enum expected
}
^
2 errors

Stephen

Java 9 EA javac exception on Windows

[Stephen Colebourne]

I ran javac on Windows and I would have expected the error to have
been caught, not to dump a stack trace. The command was to try and
compile "*.java", (which should really be implemented to find all java
files , but doesn't).

>javac --module-source-path src -d mods *.java

The exception was:

Exception in thread "main" java.nio.file.InvalidPathException: Illegal char <*>
at index 0: *.java
at sun.nio.fs.WindowsPathParser.normalize(java.base@9-ea/WindowsPathPars
er.java:182)
at sun.nio.fs.WindowsPathParser.parse(java.base@9-ea/WindowsPathParser.j
ava:153)
at sun.nio.fs.WindowsPathParser.parse(java.base@9-ea/WindowsPathParser.j
ava:77)
at sun.nio.fs.WindowsPath.parse(java.base@9-ea/WindowsPath.java:92)
at sun.nio.fs.WindowsFileSystem.getPath(java.base@9-ea/WindowsFileSystem
.java:229)
at java.nio.file.Paths.get(java.base@9-ea/Paths.java:84)
at com.sun.tools.javac.main.Option$34.process(jdk.compiler@9-ea/Option.j
ava:619)
at com.sun.tools.javac.main.Option.handleOption(jdk.compiler@9-ea/Option
.java:988)
at com.sun.tools.javac.main.Arguments.doProcessArgs(jdk.compiler@9-ea/Ar
guments.java:381)
at com.sun.tools.javac.main.Arguments.processArgs(jdk.compiler@9-ea/Argu
ments.java:303)
at com.sun.tools.javac.main.Arguments.init(jdk.compiler@9-ea/Arguments.j
ava:201)
at com.sun.tools.javac.main.Main.compile(jdk.compiler@9-ea/Main.java:212
)
at com.sun.tools.javac.main.Main.compile(jdk.compiler@9-ea/Main.java:148
)
at com.sun.tools.javac.Main.compile(jdk.compiler@9-ea/Main.java:55)
at com.sun.tools.javac.Main.main(jdk.compiler@9-ea/Main.java:41)

Re: Proposal: #ServiceLoaderEnhancements

[Stephen Colebourne]

On 13 September 2016 at 21:04,  <mark.reinh...@oracle.com> wrote:
> 2016/9/12 15:08:41 -0700, Stephen Colebourne <scolebou...@joda.org>:
>> My preference of these three options is option 2.
>
> Sorry if I wasn't clear, but this isn't meant to be a "choose one"
> proposal.  It's a set of check boxes, not radio buttons.  The proposal
> is to implement suggestions (2) and (3) of your original comment, as
> captured and labeled in the issue text, but not (1).

OK, I see. I think thats fine.

>>   However, I would
>> prefer to see the method name be defined by the module definition as
>> previously suggested [1]:
>>
>> module {
>>  provides java.sql.Driver with com.mysql.jdbc.Driver::instance;
>> }
>>
>> This would allow ::staticMethod or ::staticConstant, and avoid
>> hard-coding "provider" as a special method name.
>>
>> (For my motivating use case, the Chronology classes of ThreeTen-Extra
>> [2] it would be distracting to have a method named provider(). While a
>> separate class in a non-exported package would be possible to hold the
>> provider() method, as a solution that seems like overkill relative to
>> the syntax above.)
>
> The semantics of the `::` syntax as it already exists is tightly bound
> up with type inference.  Also, it can only be used to refer to methods,
> not fields, so it'd have to be extended to support `::staticConstant`.
>
> Working out the details is not impossible, but it's certainly far from
> trivial.  It's not clear to me that, even with more time, it should be
> at the top of our priority list.

I would be OK with just ::staticMethod. It avoids blessing a
particular method name and allows multiple methods on the same class,
but provider() is significantly better than we have today.

Stephen

Re: Proposal: #ReflectiveAccessToNonExportedTypes (revised) & #AwkwardStrongEncapsulation: Weak modules & private exports

[Stephen Colebourne]

Quite a radical proposal. Overall, I think it is better, although I do
have semantic and syntax concerns.

I'm happy that there is a way to achieve real strong encapsulation.
This seems like a good thing and will be a benefit in the long run.
For the weak modules, the proposal does not provide a way to have
packages exposed for runtime use, but not hidden at compile time.
Given the benefits of hiding internal classes from IDEs, this seems
rather unfortunate. Here is my counter proposal, hopefully a
relatively small conceptual change:

The proposal module:

  weak module foo.bar {
  // No exports
  requires hibernate.core;
  requires hibernate.entitymanager;
  }

could be written with an alternative syntax of (same semantics):

  module foo.bar {
  exports private *;
  requires hibernate.core;
  requires hibernate.entitymanager;
  }

Taking this one step further, we could separate what is exported
(public things) from what is exposed (at runtime):

  module foo.bar {
  exports *;
  exposes *;
  requires hibernate.core;
  requires hibernate.entitymanager;
  }

(again, this is the same semantics as the previous two, just different syntax)

This would then provide a straightforward way to control exports
tightly but still expose everything for reflection:

  module foo.bar {
  exports foo.bar.core;
  exposes *;
  requires hibernate.core;
  requires hibernate.entitymanager;
  }

- "exports" defines whether the public elements can be accessed
- "exposes" defines whether the package can be accessed for reflection
- there may be a one * exports clause or zero to many package name
ones, but not both types
- there may be a one * exposes clause or zero to many package name
ones, but not both types
- a package may be listed in both exports and exposes, they are orthogonal
- the "to" clause would apply to exports and exposes, but only with
package names

The above does not deviate radically from today's proposal, yet offers
a clearer syntax more power and simpler migration from weak to strong
encapsulation.

FWIW, I'm doing a presentation on modules on Wednesday, and I think
the above would be easier to explain than the "weak" keyword.

Stephen


On 12 September 2016 at 16:08, Mark Reinhold  wrote:
> Issue summary
> -
>
>   #ReflectiveAccessToNonExportedTypes --- Some kinds of framework
>   libraries require reflective access to members of the non-exported
>   types of other modules; examples include dependency injection (Guice),
>   persistence (JPA), debugging tools, code-automation tools, and
>   serialization (XStream).  In some cases the particular library to be
>   used is not known until run time (e.g., Hibernate and EclipseLink both
>   implement JPA).  This capability is also sometimes used to work around
>   bugs in unchangeable code.  Access to non-exported packages can, at
>   present, only be done via command-line flags, which is extremely
>   awkward.  Provide an easier way for reflective code to access such
>   non-exported types. [1]
>
>   #AwkwardStrongEncapsulation --- A non-public element of an exported
>   package can still be accessed via the `AccessibleObject::setAccessible`
>   method of the core reflection API.  The only way to strongly
>   encapsulate such an element is to move it to a non-exported package.
>   This makes it awkward, at best, to encapsulate the internals of a
>   package that defines a public API. [2]
>
> Proposal
> 
>
> (Warning: This is somewhat long, and in the end it affects both `exports`
>  and `requires` directives.)
>
> Extend the language of module declarations with the concept of _weak
> modules_.  Weak modules make it easy to modularize components whose
> internals will be accessed by reflection-based frameworks.  Every
> package in a weak module has the following properties:
>
>   (A) It is exported at both compile time and run time, as if by an
>   `exports` directive, and
>
>   (B) Its non-public elements are available for _deep_ reflection, i.e.,
>   at run time they can be made accessible to code outside the module
>   via the `AccessibleObject::setAccessible` method of the core
>   reflection API.
>
> In other words, every type defined in a weak module, whether public or
> not, is subject to exactly the same access checks as in Java SE 8 and
> earlier releases.
>
> A weak module is declared by placing the modifier `weak` before the
> `module` keyword.  The declaration of a weak module cannot contain any
> explicit `exports` directives.  If the `weak` modifier does not appear
> before the `module` keyword then the declared module is _strong_, and
> it can contain explicit `exports` directives.
>
> Suppose we have a module `foo.bar` which has an internal package
> `com.foo.bar.model` that contains entity classes to be manipulated by
> Hibernate, via core reflection.  Then the module declaration
>
> weak module foo.bar {
> // No 

Re: Proposal: #ServiceLoaderEnhancements

[Stephen Colebourne]

My preference of these three options is option 2. However, I would
prefer to see the method name be defined by the module definition as
previously suggested [1]:

module {
 provides java.sql.Driver with com.mysql.jdbc.Driver::instance;
}

This would allow ::staticMethod or ::staticConstant, and avoid
hard-coding "provider" as a special method name.

(For my motivating use case, the Chronology classes of ThreeTen-Extra
[2] it would be distracting to have a method named provider(). While a
separate class in a non-exported package would be possible to hold the
provider() method, as a solution that seems like overkill relative to
the syntax above.)

Stephen

[1] 
http://mail.openjdk.java.net/pipermail/jpms-spec-observers/2016-July/000535.html
[2] 
https://github.com/ThreeTen/threeten-extra/tree/master/src/main/java/org/threeten/extra/chrono


On 12 September 2016 at 16:13, Mark Reinhold  wrote:
> Issue summary
> -
>
>   #ServiceLoaderEnhancements --- The module system encourages the use of
>   services for loose coupling, but the `ServiceLoader` class is not very
>   flexible.  Consider enhancing it so that (1) neither a provider class
>   nor its no-args constructor need be declared `public`, (2) a provider
>   can be a singleton, or perhaps a collection of singletons, and (3) the
>   classes of the available providers can be inspected and selected prior
>   to instantiation. [13]
>
> Proposal
> 
>
>   (1) No change: Continue to require service-provider classes, and their
>   no-args constructors, to be public.
>
>   Providers on the class path, and their no-args constructors, must
>   always be public.  Allowing a class-path provider or its no-args
>   constructor to be non-public introduces a security risk, since an
>   adversary could place a `META-INF/services` entry elsewhere on the
>   class path in order to force that otherwise-inaccessible
>   constructor to be invoked.
>
>   For providers in named modules, allowing non-public provider
>   classes and non-public no-args constructors isn't really necessary
>   and is, in some ways, counterproductive.  In a named module a
>   provider class, and its constructor, can be encapsulated by placing
>   the provider in an unexported package.  Having to declare the
>   provider class and its no-args constructor `public` is a useful
>   declaration of intent, since they will be accessed by the service
>   loader itself, and hence useful documentation.
>
>   (2) Revise the `ServiceLoader` class so that if a candidate provider
>   class in a named module has a no-args public static method named
>   `provider` then that method is invoked and its result is taken as
>   the provider object.  An exception is thrown if the method does
>   not have an appropriate return type.  A static `provider` method
>   can either return a singleton or act as a factory method.  If the
>   candidate provider class does not have such a method then its
>   public no-args constructor, if any, is invoked, per (1) above.
>
>   (An alternative is to use an annotation, say `@Provider`, to
>identify provider-containing fields or provider-returning methods.
>The cost of loading the annotation-reading code into the JVM is,
>however, nontrivial, and since services are used widely within the
>JDK itself we'd prefer not to impose that overhead on all
>applications.)
>
>   (3) Decouple the loading of provider classes from the instantiation of
>   such classes: Introduce a new `ServiceLoader.Provider` interface
>   that pairs a provider class with a method to instantiate that
>   provider, and add a `stream()` method that returns a stream of
>   objects implementing that interface.  A client can then filter
>   providers by inspecting the elements of the stream, examining each
>   provider class and perhaps the annotations thereon, and then
>   instantiating the class if appropriate.  (Draft Javadoc source
>   attached below.)
>
> [1] 
> http://openjdk.java.net/projects/jigsaw/spec/issues/#ServiceLoaderEnhancements
>
> --
>
> /**
>  * Represents a service provider located by {@code ServiceLoader}.
>  *
>  *  When using a loader's {@link ServiceLoader#stream() stream()} 
> method
>  * then the elements are of type {@code Provider}. This allows processing
>  * to select or filter on the provider class without instantiating the
>  * provider. 
>  *
>  * @param   The service type
>  * @since 9
>  */
> public static interface Provider extends Supplier {
>
> /**
>  * Returns the provider class. There is no guarantee that this type is
>  * accessible and so attempting to instantiate it, by means of its
>  * {@link Class#newInstance() newInstance()} method for example, will
>  * fail when it is not accessible. 

Re: Closing out some open issues

[Stephen Colebourne]

On 11 September 2016 at 22:24,   wrote:
> Proposals for the following issues have been available for evaluation
> and experimentation for quite a while now.  Most responses have been
> positive and there have been no strong objections, so I've updated
> the issue list [1] to mark them as closed.
>
>   #BootstrapClassLoaderSearchInJVMTI
>   #ClassFileAccPublic
>   #CompileTimeDependences (`requires static`)
>   #CustomizableAutomaticModuleNameMapping
>   #ModuleAnnotations
>   #ModuleDeprecation
>   #ReflectiveAccessByInstrumentationAgents
>
> Not everyone was thrilled with the choice of `static` as the modifier
> on `requires` directives that indicates a compile-time dependence, but
> no obviously-better choice has emerged.

Given that "exports dynamic" has gone, there is even less reason to
use "static" (as there is no "dynamic" equivalent). The simplest
alternative is "requires optional", which fits with the existing
terminology used by maven for many years and more clearly indicates
that users cannot rely on the dependency.

Stephen

Re: Proposal: #VersionsInModuleNames

[Stephen Colebourne]

On 12 September 2016 at 16:11, Mark Reinhold  wrote:
>   - Revise the automatic-module naming algorithm implemented by `javac`
> at compile time and the `ModuleFinder::of` method [2] at run time.
> It will now strip any trailing digits and period characters that
> remain after removing the version component, if any, from the name
> of the original JAR file.  Thus `foo-bar-1.2.3.jar` becomes the
> automatic module named `foo.bar` with the version string `1.2.3`,
> and `foo-bar42.jar` becomes the automatic module named `foo.bar`
> with no version string.

I will note that this would cause problems for Apache Commons Lang.

The original release of commons uses the package
org.apache.commons.lang and has a jar file of commons-lang-2.6.jar [1]
The later release of commons uses the package org.apache.commons.lang3
and has a jar file of commons-lang3-3.4.jar [2]

Dropping the number immediately after the name will break this, and it
is not unusual for applications to have both lang and lang3 in the
classpath.

Stephen

[1] http://repo1.maven.org/maven2/commons-lang/commons-lang/2.6/
[2] http://repo1.maven.org/maven2/org/apache/commons/commons-lang3/3.4/

Re: Exporting - the wrong default?

[Stephen Colebourne]

On 2 August 2016 at 12:49, dalibor topic  wrote:
> Let's now consider two different scenarios, in which Sling and Stone are
> either public, or not, and denote the variants with a P or N prefix - PSling
> is a public Sling, while NStone is a non-public Stone, for example.
>
> Let's also consider two defaults: permissive (i.e. all packages are
> exported) and fail safe (i.e. no packages are exported).
>
> Permissive:
> PSling + PStone can induce a headache, while the other three cases
> fortunately can not. So a mere 25% opportunity for a headache.
>
> Fail safe:
> none of the four cases induce a headache.

Er, now no-one can use the library.

The purpose of a library is to provide public types for applications
to use. That these have bugs and/or security issues is a fact of life
in software. Developers are not going to suddenly stop using external
OSS libraries.

> Let's now add reflection & setAccessible to the mix.
> ...
> Again, no headache occurs in the Failsafe case, because no packages are
> exported,

Er, now no-one can use the library because it can't operate without
reflection+setAccessible.

I again draw attention to the fact that many many OSS libraries use
reflection+setAccessible, and that the ecosystem relies on it pretty
much everywhere [1], at least in part because beans+properties has
never been addressed.

> Of course, you can get to the same cases starting with a Permissive default
> and then restricting exports of the packages in question.
>
> Yet, there is an interesting difference between the two scenarios:
>
> a) in the fail safe default case you start at no risk of headache inducing
> combinations of Slings and Stones, and then can increase your risk of a
> headache occurring to a certain level by adding exports, while in the
> permissive case you start at a coin toss level of headache risk and have to
> remove exports to decrease it to the same level, and
>
> b) regardless of how many Slings and Stones there are in the mix, you always
> start at no risk of headache inducing combinations of Stones and Slings in
> the fail safe default case, while adding further Slings and Stones to the
> mix makes your initial headache risk significantly greater than a coin toss
> in the permissive case.

Er, but this is such a straw man. There is no "initial headache risk".
The packages have to be exported to actually use the library, and are
today. The vast majority of libraries porting to Java 9 will have to
export all packages because nothing else will work. In virtually no
cases, is the end-user developer going to be changing the exported
packages of the libraries they depend on.

To get any benefit from the new modular scope, existing OSS projects
will require significant backward-incompatible reworking - to take
existing public classes and methods and move them to different
packages. This is major work, and far less likely than just exporting
everything (how many years has it taken to do this in the JDK?).

What I want from jigsaw is a *new* ability, not alteration of the
status quo. ie. to allow developers to *opt-in* to selectively hide
things at the module level *going forward*. Ideally I also want to
hide individual classes, methods and fields, because packages are far
too coarse grained for this.

IMO, we need to get existing OSS libraries migrated to modules ASAP to
maintain the ecosystem, which requires the migration to be really
simple. Get this wrong, and we'll all just carry on using the
classpath, and the last 5+ years of effort in the JDK will have little
value.

Stephen

[1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008855.html

Re: Exporting - the wrong default?

[Stephen Colebourne]

On 28 July 2016 at 15:47, Russell Gold  wrote:
> Did you show this in an article we can access?

On this list:
http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008823.html

Stephen

Re: Exporting - the wrong default?

[Stephen Colebourne]

This artificial description/analogy seems to bear no relation to real
world development or the proposal in this thread,

To get any module to work with the basic jigsaw design, it has to
somehow expose a set of packages. As I have shown, in most libraries,
90%+ of the packages need to be exposed - hiding packages is the
minority/specialist use case (the JDK is unusual in the packages it
wants to hide).

This proposal simply alters how that set of packages is calculated
when processing module-info. It asks the module writer to deliberately
hide something, rather than to slavishy create a super long list of
exposed packages. But the binary format could still list the exported
packages rather than the restrictions.

No more packages would be exposed than with the current proposal. No
more headache inducing problems would be created.

Stephen


On 28 July 2016 at 13:33, dalibor topic  wrote:
> In the "exported by default" world view, the assembler is responsible for
> restricting all such headache inducing interactions between these classes,
> originating in different components with different trade offs.
>
> Let's hope for their sake they are really awesome experts at that sort of
> thing, and especially good at handling the potentially exponential
> complexities that can arise from adding new components with further third
> party provided Guns and Bullets to their software system.
>
> In the "failsafe by default" world view, they are responsible for enabling
> "just" the non-headache inducing interactions between those classes, of
> which there in most cases may very well be none.
>
> That's (potentially) a substantial difference in effort necessary to
> accurately make such trade offs, in particular over the maintenance life
> cycle of a software system.

Re: Exporting - the wrong default?

[Stephen Colebourne]

On 27 July 2016 at 16:26, Remi Forax  wrote:
> to get back to our issue,
> there are 4 possibilities when exporting a package, for a public type,
> (1) don't see it at compile time, don't see it at runtime (can't reflect on 
> it)
> (2) don't see it at compile time, see it at runtime (this is the OSGI/JBoss 
> model for not exported)
> (3) see it at compile time, may not exist at runtime (so be prepared to get 
> an exception then)
> (4) see it at compile time and see it at runtime

Agreed

> The default can not be (3) because it's a corner case,

Agreed

> it can not be (4) because in that case we lost the 'strong encapsulation' 
> that a module should provide by default,

That is what this thread discusses. It seems to me that the "strong
encapsulation" goal is met providing that a package can be hidden if
desired, and that the set of packages a module exports is still known
and limited (just automated by the compiler). Option (4) also
mitigates the issue that David Holmes has repeatedly indicated, where
jigsaw is currently planning on changing the meaning of "public".

The key point is that because modules are being added to Java late,
the only appropriate design is for them to be easily opt-in. While the
module system goes a fair way towards that goal, it would be further
aided by specifying packages to hide, rather than packages to export.

I agree with the rest of the mail, notably that unless we get this
right, there will be very little incentive to use the module system in
open source or applications.

Stephen


> so the default can be either (1), either (2) or to force the user to choose 
> between (1) and (2) when declaring a module.
>
> The problem with (1) is that:
>  - it makes most of the code that use reflection not working (and as Stephen 
> said, at lot of codes use reflection (or bytecode generation)),
>  - it will slow down the adoption of jigsaw (not jdk9 which will be run with 
> a -classpth) but the modularization of the already existing jars, so we will 
> end up with a module system which will be not used or worst, some jars will 
> be modularized, some will not and we will be in the same sad state of Python 
> now with 2 mostly compatible worlds *.
>
> The problem of letting users to choose is that the hope to educate them by 
> forcing them to make their own choices will be destroyed because in practice 
> IDEs will chose for them (e.printStackTrace() anyone ?)
>
> So the only valid choice seem to be (2), which
> - still enable JDK and application server implementation modules to not 
> export some types at runtime, so the security will improve and by example, it 
> will avoid most of the access control bugs Christina talk about.
> - the default behavior will make the move to convert their jars to 
> modularized jars easier because people will not conflate the problem of the 
> modularization itself with the problem of the access control at runtime.
> - everybody will be happy and we will not see angry ponies on slides about 
> Java 9.
>
>>
>> cheers,
>> dalibor topic
>
> cheers,
> Rémi
>
> * Or, at some point, someone will also find that by using jlink and creating 
> its own module Layer, he can have a 'Java' launcher with its own defaults.

Re: Exporting - the wrong default?

[Stephen Colebourne]

The purpose of this change is meant to be to enhance Java. While
security may be part of the issue being tackled, it has never been
claimed as the main one. This proposal does not make it much of an
issue either, as the runtime would likely still have a list of
exported packages - it would be the declaration format
(module-info.java) that would change. (javac has a closed set of
packages when compiling a module, so outputting exports in the binary
but using restricts in module-info works just fine)

I think a key problem is that those working on the JDK have a warped
sense of what a typical Java project is like. In many projects
packages change names frequently during development, everything is
open and locking stuff down is the last thing on peoples minds. While
this of course leads to slightly less secure software, it does achieve
*business value*. The current default to force developers to list all
exports is simply going to slow developers down for no perceived
benefit. (In fact with Java EE and OSGi excluded from modules, they
really don't seem all that useful anyway).

The "robust in the face of change" argument is dubious too. If you
want package exports to be robust in the face of change, the
export/restruction criteria should be in packag-info.java, not
module-info.java, where it would automatically be handled by all
existing tooling for renaming packages.

If the default does not change, then I assume the tools will
effectively work around it. ie. that Maven will have an option to
export all packages not explicitly hidden. Anything else would be far
too annoying for development.

Stephen
PS, I would accept a design where developers could choose to use
exports or restricts (but not both) in module-info.java.


On 26 July 2016 at 11:48, dalibor topic <dalibor.to...@oracle.com> wrote:
> On 26.07.2016 12:30, Stephen Colebourne wrote:
>>
>> This does not appear to change the underlying model of modules
>> (reliable configuration and strong encapsulation), but would make it
>> much more practical to use.
>
>
> It wouldn't be as robust in face of change, as it would require consciously
> tracking new packages being added to a module and explicitly marking them as
> internal, or living with design mistakes (or other problems) because someone
> forgot to immediately restrict access to something that was supposed to be
> internal.
>
> For a much longer explanation, see "Fail-safe defaults" in
> https://buildsecurityin.us-cert.gov/articles/knowledge/principles/failing-securely
>
> cheers,
> dalibor topic
> --
> <http://www.oracle.com> Dalibor Topic | Principal Product Manager
> Phone: +494089091214 <tel:+494089091214> | Mobile: +491737185961
> <tel:+491737185961>
>
> ORACLE Deutschland B.V. & Co. KG | Kühnehöfe 5 | 22761 Hamburg
>
> ORACLE Deutschland B.V. & Co. KG
> Hauptverwaltung: Riesstr. 25, D-80992 München
> Registergericht: Amtsgericht München, HRA 95603
>
> Komplementärin: ORACLE Deutschland Verwaltung B.V.
> Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
> Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
> Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher
>
> <http://www.oracle.com/commitment> Oracle is committed to developing
> practices and products that help protect the environment

Re: Using modules in open source libraries

[Stephen Colebourne]

Exported vs non-exported determined by looking at the packages and
deciding if end users need to be able to compile against them.
Stephen


On 26 July 2016 at 11:48, Alessio Stalla <alessiosta...@gmail.com> wrote:
> Just curious - how did you identify exported and non-exported packages? Are
> you speaking about your design intentions or are you looking at actual use
> of the libraries in wild?
>
> On 26 July 2016 at 12:10, Stephen Colebourne <scolebou...@joda.org> wrote:
>>
>> There has been a lot of discussion about exporting packages - at
>> compile time vs runtime and also the related problem of resources. I
>> decided to take a look at some open source projects to see whether a
>> list of exported packages is useful as currently defined:
>>
>> These standard open source projects were designed prior to modules:
>>
>> Joda-Time - 7 exported packages
>> Joda-Time-Hibernate - 1 exported package
>> Joda-Time-I18N - 1 exported package
>> Joda-Time-JspTags - 1 exported package
>> Joda-Money - 2 exported packages
>> Joda-Primitives - 10 exported packages
>> Joda-Convert - 2 exported packages
>> Joda-Collect - 1 exported package
>> Joda-Beans - 16 exported packages
>> Joda-Beans-Maven-Plugin - 1 exported package
>> Joda-Beans-UI - 5 exported packages
>> ThreeTen-Extra - 3 exported packages
>> ThreeTen-Backport - 6 exported packages
>> Google Guava - 18 exported packages, 1 non-exported package
>> SLF4J - 3 exported packages
>> JCommander - 4 exported packages, 1 non-exported package
>>
>> Looking at the list, it is clear that non-exported packages are a
>> small use-case.
>> - 81 exported packages
>> - 2 non-exported packages
>>
>> Strata [1] has been designed in Java 8 but with an awareness that
>> modules are coming:
>>
>> Strata-Collect - 8 exported packages
>> Strata-Basics - 7 exported packages
>> Strata-Data - 2 exported packages
>> Strata-Calc - 3 exported packages
>> Strata-Product - 23 exported packages
>> Strata-Market - 13 exported packages
>> Strata-Loader - 3 exported packages, 1 only exported at runtime for
>> reflection
>> Strata-Math - 1 exported package, 17 packages exported only to a
>> single friend module
>> Strata-Pricer - 19 exported packages, 10 non-exported packages
>> Strata-Measure - 15 exported packages
>>
>> - 94 exported packages
>> - 17 exported to a single other module
>> - 11 non-exported packages
>>
>> I'll use a different thread to express my opinion on what the data
>> means. Please use this thread to add summaries of other open source
>> projects.
>>
>> Stephen
>>
>> [1] https://github.com/OpenGamma/Strata
>
>

Exporting - the wrong default?

[Stephen Colebourne]

Over recent weeks, there has been a long debate about exporting at
runtime, with other discussions about resources. I want to express my
view that listing specific packages for export seems to be the wrong
approach.

In my last thread [1], I found that just 2.5% of packages in the open
source libraries I looked at were suitable for being non-exported. In
the Strata library, designed for modules, the number rose to 11.7%.

As such, shouldn't we consider that the default for modules should be
changed? Instead of specifying the list of packages that are to be
exported, the module system should specify the list of packages that
are to be restricted:

module com.foo.bar {
requires org.baz.qux;

restricts com.foo.bar.alpha;  // package is not exported
restricts com.foo.bar.beta to com.foo.baz;  // export only to a friend
restricts at compiletime com.foo.bar.beta;  // export "dynamic"
}

This does not appear to change the underlying model of modules
(reliable configuration and strong encapsulation), but would make it
much more practical to use. After all, isn't Java meant to be designed
for the 80% use case? (which is clearly that everything is exported in
most projects)

Stephen

[1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008823.html

Re: Producing "bundled" JREs for other platforms?

[Stephen Colebourne]

Just to note that we, at OpenGamma, had this requirement too (in the
past, not now). ie. the requirement to have a single jar file that
contains the necessary binary parts to run on Windows, Linux and Mac.
The jmod stuff ought to be designed to support this, as while the
requirement is rare, it is complex to get right, The code we used
(before development stopped) is here -
https://github.com/OpenGamma/OG-Maths - which produces a jar file with
Java code plus native code for all three platforms.

Stephen


On 24 July 2016 at 22:06,   wrote:
> 'Lo.
>
> On 2016-07-24T21:50:48 +0100
> Alan Bateman  wrote:
>> jlink produces a run-time image for a specific platform. There are a few
>> rough edges when it comes to cross targeting but it should work, you
>> just specify the location of the packaged modules for the target platform.
>
> Got it.
>
>> However if read your mail correctly then you seem to be looking for a
>> "universal binary". I can't quite tell if you are looking to
>> redistribute a run-image or not, the above suggests you must be picking
>> that up from whatever is on the platform already.
>
> I'm not entirely sure what I'm looking for right now, just looking at
> what Jigsaw provides, really.
>
> My overall goal with distributing software that I've written is to have
> one single distributable package that runs on as many platforms as
> possible. I suppose that does sort of fit the notion of a "universal
> binary".
>
> The arrangement that I described in the previous email is good in the
> sense that 99% of the distributed package is platform-independent code
> that's shared between all supported platforms. The platform-specific
> parts are limited to tiny launcher executables. Even better, if I
> haven't provided any sort of launcher for a platform, a technologically
> competent user on that platform can get my program to run with a
> little work: I'm providing the code as platform-independent jar files
> and the user can simply pass them to whatever JVM they're using.
>
> From what I can see, Jigsaw's images would actually be a step backwards
> here. It seems like I would still need to provide platform-specific
> launcher executables, and even worse, the very process of creating
> images would eliminate any sharing of compiled code between platforms.
> I could distribute a single package that contains images for different
> platforms, but it would contain a lot of duplicated compiled code and
> would effectively limit the package to running on only those platforms
> for which images have been produced.
>
> M

Re: modulepath and classpath mixture

[Stephen Colebourne]

On 30 March 2016 at 14:45, Alan Bateman  wrote:
> The thread here has meandered a bit but I think the scenario under
> discussion is tests for a module that need to nestmate with the module under
> test. The tests are in their own test tree. The tests are compiled
> separately from the module they test and may have additional dependences
> (such as on TestNG or JUnit for example). When compiling or running then the
> tests need to access public types in non-exported packages and maybe package
> private members too.

Yes, plus there is a need for one set of  tests to depend on another
set of tests - ie. tests can be an artifact.

> The support for this has been in jake for a long time
> but involves command line options that many developers or build environments
> won't immediately grok. In particular the tests have to be compiled "as if"
> they augment the already compiled module - that is what javac -Xmodule is
> about. There is no need to co-locate source files or class files of course.
> When run then the -Xpatch option is what brings the tests and the module
> classes together. If we get the tools right then most developers won't ever
> see this of course.

This is perhaps a solution that works, but it comes across as highly
esoteric. Two different command line flags, augmenting/patching
modules. What I see is a nightmare to explain.

And while tools can make things easy, it is inadvisable for the design
for running something as basic as tests to be so complex that it
cannot practically be done without tools.

What I see as a better solution is the same approach as I argued for
with optional dependencies - enhancing the syntax of module-info.java
to avoid the need for command line flags

Stephen

Re: modulepath and classpath mixture

[Stephen Colebourne]

On 29 March 2016 at 17:33, Russell Gold <russell.g...@oracle.com> wrote:
> Hi Stephen,
> Why do you need any kind of friend access?
> It seems to me that this is making things harder than they need to be. The 
> tests can simply run with the production code on the class path, and then 
> there are no module issues at all.

The underlying message of Jigsaw is that the classpath is going away.
An approach that requires use of the classpath is therefore not really
a solution to the real problem.

Stephen


> I think a larger problem is that you can do what I just said with the jars, 
> even a jar which has been designated as a module by virtue of having a 
> module-info.class in it. That means that, when users are up taking jars, they 
> are not prevented from accessing module internals. They can put the jars on 
> the module path, of course, but they can still use them on the class path!
>
> - Russ
>
>>
>> On 28 March 2016 at 11:13, Remi Forax <fo...@univ-mlv.fr> wrote:
>>> Hi Stephen, Hi all,
>>> I think that delivering tests as a separated module is a bad idea.
>>>
>>> I see that from the point of a developer, seeing the code and the test as 
>>> different modules can be attractive because everything seems to be put at 
>>> the right place but there is a big drawback. Because modules are reified at 
>>> runtime, it means that the runtime environment of the tests will be 
>>> different from the production environment.
>>
>> This last sentence doesn't make sense to me - tests are not run in a
>> production environment.
>>
>> Tests have all the qualities of modules - code, dependencies,
>> compilation phase, deployment. The only special part is the need for
>> special "friend-like" access, which Jigsaw already has for other cases
>> (the export...to clause).
>>
>> Put simply, I consider that module =
>> deployment-artifact-with-dependencies. With that mental model, putting
>> tests inside the module is just not acceptable, because tests should
>> not be deployed with the main application and they have different
>> dependencies. If we disagree that module =
>> deployment-artifact-with-dependencies, then perhaps we have bigger
>> problems to solve here.
>>
>> Stephen
>> (And to Paul Benedict, the classpath is going to die over time, so any
>> solution that uses that is flawed IMO).
>>
>>
>>> So as Alan said, from the jigsaw point of view at runtime, the tests and 
>>> the code should be in the same module.
>>>
>>> So the building tools have to come with a way to support to have 2 
>>> different module-info.java in two different folders and package them as one 
>>> module,
>>> maybe javac should help by providing a way to merge 2 module-info at 
>>> compile time.
>>>
>>> Rémi
>>>
>>> - Mail original -
>>>> De: "Alan Bateman" <alan.bate...@oracle.com>
>>>> À: "Stephen Colebourne" <scolebou...@joda.org>, "jigsaw-dev" 
>>>> <jigsaw-dev@openjdk.java.net>
>>>> Envoyé: Mercredi 23 Mars 2016 16:18:50
>>>> Objet: Re: modulepath and classpath mixture
>>>>
>>>>
>>>> On 23/03/2016 14:42, Stephen Colebourne wrote:
>>>>> :
>>>>>
>>>>> I don't particularly care what the mechanism is for this, but at the
>>>>> requirements level:
>>>>> - there are two modules - main and test
>>>>> - each has its own source tree
>>>>> - each has its own dependencies
>>>>> - each is released separately
>>>>> - each could be hosted on a central repo
>>>>> - the test module needs to be able to contain the same packages as the
>>>>> main module
>>>>> - the test module needs to be able to invoke package-scoped code in
>>>>> the same package in the main module
>>>>>
>>>>> To clarify further consider 4 modules, A, B, A-test and B-test where B
>>>>> depends on A. Module A-test may have a method foo() that uses package
>>>>> scope to access something in A. Module B-test will depend on A-test
>>>>> and rely on foo() to get access to that internal object.
>>>> To your list, I would add the ability to make use of testing frameworks
>>>> like TestNG and JUnit.
>>>>
>>>> In any case, and as things currently stand, you've got most of the
>>>> above. One differences is that the tests are not a separate module, they
>>>> are instead compiled and run in a way that patches the main module. The
>>>> second difference is that they don't have their own module declaration,
>>>> instead the compilation or run augments the dependences with any
>>>> additional dependences that the tests have. As I said, if they tools
>>>> makes it easy then I don't think it's too bad.
>>>>
>>>>>
>>>>> (Note that I view the thread discussion of
>>>>> references to test classes on the classpath as another hack.
>>>>>
>>>> Packages can't be split between modules and classpath so there is no
>>>> support for that.
>>>>
>>>> -Alan
>>>>
>

Re: modulepath and classpath mixture

[Stephen Colebourne]

On 28 March 2016 at 11:13, Remi Forax <fo...@univ-mlv.fr> wrote:
> Hi Stephen, Hi all,
> I think that delivering tests as a separated module is a bad idea.
>
> I see that from the point of a developer, seeing the code and the test as 
> different modules can be attractive because everything seems to be put at the 
> right place but there is a big drawback. Because modules are reified at 
> runtime, it means that the runtime environment of the tests will be different 
> from the production environment.

This last sentence doesn't make sense to me - tests are not run in a
production environment.

Tests have all the qualities of modules - code, dependencies,
compilation phase, deployment. The only special part is the need for
special "friend-like" access, which Jigsaw already has for other cases
(the export...to clause).

Put simply, I consider that module =
deployment-artifact-with-dependencies. With that mental model, putting
tests inside the module is just not acceptable, because tests should
not be deployed with the main application and they have different
dependencies. If we disagree that module =
deployment-artifact-with-dependencies, then perhaps we have bigger
problems to solve here.

Stephen
(And to Paul Benedict, the classpath is going to die over time, so any
solution that uses that is flawed IMO).


> So as Alan said, from the jigsaw point of view at runtime, the tests and the 
> code should be in the same module.
>
> So the building tools have to come with a way to support to have 2 different 
> module-info.java in two different folders and package them as one module,
> maybe javac should help by providing a way to merge 2 module-info at compile 
> time.
>
> Rémi
>
> - Mail original -
>> De: "Alan Bateman" <alan.bate...@oracle.com>
>> À: "Stephen Colebourne" <scolebou...@joda.org>, "jigsaw-dev" 
>> <jigsaw-dev@openjdk.java.net>
>> Envoyé: Mercredi 23 Mars 2016 16:18:50
>> Objet: Re: modulepath and classpath mixture
>>
>>
>> On 23/03/2016 14:42, Stephen Colebourne wrote:
>> > :
>> >
>> > I don't particularly care what the mechanism is for this, but at the
>> > requirements level:
>> > - there are two modules - main and test
>> > - each has its own source tree
>> > - each has its own dependencies
>> > - each is released separately
>> > - each could be hosted on a central repo
>> > - the test module needs to be able to contain the same packages as the
>> > main module
>> > - the test module needs to be able to invoke package-scoped code in
>> > the same package in the main module
>> >
>> > To clarify further consider 4 modules, A, B, A-test and B-test where B
>> > depends on A. Module A-test may have a method foo() that uses package
>> > scope to access something in A. Module B-test will depend on A-test
>> > and rely on foo() to get access to that internal object.
>> To your list, I would add the ability to make use of testing frameworks
>> like TestNG and JUnit.
>>
>> In any case, and as things currently stand, you've got most of the
>> above. One differences is that the tests are not a separate module, they
>> are instead compiled and run in a way that patches the main module. The
>> second difference is that they don't have their own module declaration,
>> instead the compilation or run augments the dependences with any
>> additional dependences that the tests have. As I said, if they tools
>> makes it easy then I don't think it's too bad.
>>
>> >
>> > (Note that I view the thread discussion of
>> > references to test classes on the classpath as another hack.
>> >
>> Packages can't be split between modules and classpath so there is no
>> support for that.
>>
>> -Alan
>>

Re: modulepath and classpath mixture

[Stephen Colebourne]

On 23 March 2016 at 12:51, Alan Bateman  wrote:
> If types T1 and T2 have the same defining loader and both types are in the
> same package then they are in the same module. T1 can't be module M1 and T2
> in a different module M2.
> (I shudder the thought of it being different or attempting to change what
> default/package access means after all these years.)

I'm not asking for package scope to change meaning. It is Jigsaw that
is trying to change its meaning wrt tests.

Discussion of white vs black box testing is all very well and good,
but its completely impractical. There are tens of thousands of
existing projects out there, in open source and private, that rely on
tests being in the same package. It is in essence a fundamental part
of common approaches to testing. Making it hard to test code in the
same package, as seems to be proposed, would be a nightmare for
migration. (Many open source projects are for fun in spare time. Who
is going to want to refactor all their tests to meet what amounts to
an artificial limitation? Not me.)

I don't particularly care what the mechanism is for this, but at the
requirements level:
- there are two modules - main and test
- each has its own source tree
- each has its own dependencies
- each is released separately
- each could be hosted on a central repo
- the test module needs to be able to contain the same packages as the
main module
- the test module needs to be able to invoke package-scoped code in
the same package in the main module

To clarify further consider 4 modules, A, B, A-test and B-test where B
depends on A. Module A-test may have a method foo() that uses package
scope to access something in A. Module B-test will depend on A-test
and rely on foo() to get access to that internal object.

One apparent option would appear to be to merge the two modules
dynamically at load time. However, they do need to be released and
published separately (Note that I view the thread discussion of
references to test classes on the classpath as another hack. The
release and dependency elements clearly show tests to be modules, just
ones with special access.)

Stephen

Re: modulepath and classpath mixture

[Stephen Colebourne]

On 23 March 2016 at 07:21, Alan Bateman  wrote:
> If they are in the same class loader and package as
> the code they are testing (the norm in Maven) then they need to compiled as
> if they are part of the module.

I struggle with that idea.

To me, tests are very definitely not part of the module. To me, they
are very clearly a separate module, one that uses the base module
(separate tree of code, separate dependencies, separate compilation
phase). In maven, there is even the possibility for one testing module
to depend on another testing module, a very useful feature.

The only special case about testing modules is that they are given
full access to everything in the underlying module. Forcing test code
to be embedded within the actual module just because the module design
doesn't allow any other option is flawed IMO.

Peter Levart's mail suggested one approach to provide the necessary
access rights for testing modules. But it still requires publication
package-by-package. Really, what is needed is a way to express that
the testing module is intimately connected, perhaps imposing a
requirement that both are loaded in the same classloader. eg.

module my.mod {
fully-exposed-to my.mod.test;
}
module my.mod.test {
requires my.mod;  // since other end specifies fully-exposed-to,
this gets full access
requires junit;
}

Is this issue on the list of open issues?
http://openjdk.java.net/projects/jigsaw/spec/issues/

Stephen

Re: Automatic module names

[Stephen Colebourne]

On 10 March 2016 at 07:58, Alan Bateman  wrote:
> Keep in mind that a compiled module-info can be added to a JAR file where
> the code is otherwise compiled for an older release. If you are willing to
> tolerate a newer JDK in your build environment then you can compile that
> code with `javac -release N`(see JEP 247 [1]). This means one JDK in your
> build environment with the build creating a JAR file that you can deploy on
> the class path with JDK N or newer, or as an explicit module with JDK 9 or
> newer. It is of course a more complicated build but something that a
> "forward looking" developer could do.

Joda-Time compiles for JDK 1.5, other Joda projects for 1..6. Last
time I looked, the tool didn't go back far enough to be useful.

Stephen

Re: Missing issue? - indexing

[Stephen Colebourne]

On 2 March 2016 at 13:30, Alan Bateman  wrote:
> Just on existing tools that are scanning the class path, have you run into
> issues there that suggest they won't work as before?

I know that the internals of reflections are complex, trying to find
files in jars, classpath and other app-server specific locations. I
haven't tried it on JDK 9, but I'd imagine it would need alteration.
Stephen

Re: Spring's need for optional dependencies

[Stephen Colebourne]

And here are the threads for Joda projects, which also need optional
dependencies:
http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005462.html
http://mail.openjdk.java.net/pipermail/jigsaw-dev/2015-December/005638.html

Note, I do not consider command line flags to be acceptable as a solution.

Stephen


On 17 December 2015 at 09:41, Stephane Epardaud  wrote:
> As I already mentioned, we also have the need for this in Ceylon, for
> the same reasons. Dependencies are required at compile-time but optional
> at run-time, based on detection: if it's there fine, if not then no problem.

Re: Optional dependencies

[Stephen Colebourne]

On 5 December 2015 at 09:49, Alan Bateman  wrote:
>
> On 04/12/2015 19:10, Peter Levart wrote:
>>
>> :
>>
>> You would have to invoke jodaBeansModule.addRead(guavaModule); from your
>> org.joda.beans code before 1st use of any guava type. In Addition, you would
>> have to ensure guava is part of runtime configuration - either implicitly by
>> some module in your configuration requiring guava or explicitly by adding
>> "-addmodule guava" to the java launcher command line.
>
> Yes, that's right, there is enough to support all the scenarios. The
> use-guava-if-it-happens-to-be-there scenario is somewhat awkward though as
> it would means using -addmods. On the other hand, the
> use-guava-if-someone-hands-me-a-guava-type scenario is straight-forward, it
> just means reading the module at run-time (easy).
>
> -Alan

Re: Optional dependencies

[Stephen Colebourne]

On 5 December 2015 at 09:49, Alan Bateman  wrote:
> Yes, that's right, there is enough to support all the scenarios. The
> use-guava-if-it-happens-to-be-there scenario is somewhat awkward though as
> it would means using -addmods. On the other hand, the
> use-guava-if-someone-hands-me-a-guava-type scenario is straight-forward, it
> just means reading the module at run-time (easy).

However taking a step back, in order to support this type of release,
command line flags are not an option. Since Joda-Beans is an OSS
library, developers need to be able to "just use it". Telling
developers that if they use this library they have to add a complex
command line flag at runtime is simply not on.

To me, the answers have identified a clear need for optional module
dependencies in the jigsaw design, such that key information is not
lost.

Stephen

Re: Optional dependencies

[Stephen Colebourne]

I've been pondering whether -addReads is sufficient for this use case.

While type 1 of my classification below (annotations) would work using
compile-time -addReads, I don't think types 2 and 3 would. (Type 2 is
reflection based, type 3 is usage based)

Here is the setup:

module user {
  requires org.joda.beans
  requires com.google.guava
}
module org.joda.beans {
  // optionally requires com.google.guava
}

Proposed solution:
compile module with -addReads=com.google.guavea

The proposed solution would allow the Joda-Beans module to be
successfully compiled. But it seems that it would not be able to see
the Guava code at runtime (as the module does not have the
dependency). Note that there is source code in Joda-Beans that
actively calls Guava clases in addition to checking they exist by
reflection.

Given the above setup, it seems to me that -addReads at compile time
is insufficient to meet the requirement of optional dependencies.

IMO, adding syntax to the module-info for optional dependencies would
allow the connection between the modules to be recorded properly. With
that information, the module system could grant the implied
readability link from Joda-Beans to Guava provided that some other
module has Guava as a dependency.

Stephen



On 1 December 2015 at 15:49, Stephen Colebourne <scolebou...@joda.org> wrote:
> On 1 December 2015 at 15:39, Alan Bateman <alan.bate...@oracle.com> wrote:
>> This should be doable but maybe not obvious. Can you say a bit more about
>> the Joda-Time optional dependency on Joda-Convert first? In particular, are
>> there static references to types in Joda-Convert and maybe a reflection
>> guard to check the presence of one of its types?
>>
>> It might be that you have to compile module joda.time with
>> -XaddReads:joda.time=joda.convert. At run-time then you can use Module
>> addReads to read module joda.convert when it's in the module graph.
>
> So, there are three variants of dependency in that list.
>
> 1) Joda-Time depends on Joda-Convert solely for annotations. Thus, the
> annotations need to be available to compile against. The annotations
> have runtime scope, so must remain in the bytecode. However, only
> those users that want to use Joda-Convert in anger will ever care
> about them. For everyone else, the fact that there is no class file
> for the annotation at runtime is not a problem.
>
> 2) The Joda-Convert dependency on Guava is reflection based. If it
> finds the class on the classpath it will enable the conversion logic.
> If it doesn't it won't.
>
> 3) The ElSql dependency on Spring is not reflection based. If the user
> uses the class ElSqlBundle, they have to have Spring available. If
> they do not use that class, they do not have to have Spring available.
>
> The Joda-Beans dependencies are a mixture of the second and third type..
>
>
> I can see how the compile time flag could work, which is good.
> However, I think that it should be part of the module-info file, since
> the project cannot be compiled without it.
>
> module org.joda.time {
>   exports ...
>   at-compile-time {
> requires module org.joda.convert;
>   }
> }
>
> Stephen

Feature complete?

[Stephen Colebourne]

According to the schedule, JDK 9 is meant to be feature complete in 9 days.
http://openjdk.java.net/projects/jdk9/
Without casting any aspersions on the hard work going on, it seems to
me that Jigsaw is nowhere near feature complete.

A recent mail from David M. Lloyd to the spec mailing list raises
serious concerns about the current design, yet it has not been
answered by the Jigsaw team.
http://mail.openjdk.java.net/pipermail/jpms-spec-observers/2015-November/000225.html

Nor has there been much activity on the spec list in general:
http://mail.openjdk.java.net/pipermail/jpms-spec-observers/2015-November/thread.html

I personally am far from convinced that the compatibility of JDK 9
with 8 is going to be good enough. Especially as it seems to me that
all libraries that use reflection are likely to be broken with 9,
something that will significantly hamper adoption.

Can we have a formal view on whether Jigsaw is considered "almost
complete"? To me, it looks like it needs a fair bit longer to bake.

Stephen

Re: Feature complete?

[Stephen Colebourne]

On 1 December 2015 at 13:24, Alan Bateman  wrote:
> If your comment is more about the challenge that some frameworks will have
> with strong encapsulation then it's a fair concern. As things currently
> stand then there isn't a mechanism (aside from the command line) to give
> specific frameworks the power to break encapsulation and get at the
> internals of other modules. So definitely some TBD here, maybe on the
> framework side too because modules might provide an opportunity to revisit a
> few things.

The JavaOne talks specifically mention the need for code changes for
reflection code (adding readability IIRC). And I know there will be
lots of psuedo code that says:
if (!member.isPublic) {
  member = member.setAccessible(true)
}
which is also likely to have problems, because public no longer has
exactly the same meaning as today.
Stephen

Re: Annotations across modules

[Stephen Colebourne]

On 16 November 2015 at 22:04, Alex Buckley  wrote:
> This is all true -- a static reference to java.beans.ConstructorProperties
> necessitates a dependency on java.desktop at compile time.
>
> The reason for my stripped-down answer was that Stephen phrased the question
> using some rather abstract language -- "an annotation is placed into a
> class", "a user may run my software ...". This made me think something
> clever was going on -- for example, the user's dependency on java.desktop is
> factored out and Stephen's software receives a Class object for
> java.beans.ConstructorProperties from someone else.

FWIW, the case I was considering no longer applies. Joda-Beans is a
source code generator. I was planning on always adding
@ConstructorProperties and relying on the fact that if the annotation
is not found at runtime it does not matter. But the discussion above
indicates that it will be hard to have a different set of modules at
compile time to runtime. In the end, I made generation of
@ConstructorProperties opt-in, so the problem effectively goes away.

I suspect that I will return to discuss optional dependencies before
too long, as it is a big concern of mine.

Stephen

Annotations across modules

[Stephen Colebourne]

My understanding of annotations today is that annotations are not
required at runtime. ie. if an annotation is placed into a class at
compile time but the .class file for the annotation is not available
on the runtime classpath. then there is no error.

Does this change in any way with modules?

My specific case is wrt @ConstructorProperties which is in java.beans.
In 9, a user may run my software without the java.beans module. Can I
safely assume that if the annotation is available on the compile
module path but not the runtime module path, everything will be OK
(ie. no error)?

thanks
Stephen

Re: Avoiding same-package conflicts

[Stephen Colebourne]

On 29 October 2015 at 13:48, Alan Bateman  wrote:
> The restriction is that no two modules with the same package (irrespective
> of whether it is exported or not, or who reads the modules) can be mapped to
> the same class loader.

FWIW, I think that this could be a headache for module adoption.

Here are three cases that appear to be troubled by these restrictions:

- a large project that has taken an existing project (module) and
split it in two. In order to preserve backwards compatibility, the
author wants to retain the package names. But doing so is not
possible, because a single package now needs to be split across two
modules.

- extending a package scoped API from another project. While hardly
pleasant this is sometimes the only viable approach to get a task
done. As an example, the ImmutableList class from Google Guava has a
package scoped constructor. It is perfectly possible today for an
application to write a class that extends ImmutableList, so long as
that class is placed in the com.google.common.collect package. The
module system has the potential to block this extension point. (I
imagine that cglib and similar tools also generate code in other
peoples modules)

- where a stable interface is copied into two separate jar files to
avoid a dependency. An example of this was commons-beanutils where 4
interfaces were copied from commons-collections. While this has now
been changed, it was considered useful at the time, and given the
stability of the interfaces in question, caused no problems.
https://issues.apache.org/jira/browse/BEANUTILS-379

FWIW, I'm most concerned about the first one going forward. However,
the other two are cases I've seen used, and if used broadly in the
wild could slow adoption (if it is hard to make those jar files
modular, and they are widely depended on).

Stephen

Re: Compilation feedback and version question

[Stephen Colebourne]

On 28 October 2015 at 23:59, Alan Bateman <alan.bate...@oracle.com> wrote:
> On 28/10/2015 20:50, Stephen Colebourne wrote:
>> Just to note that my experiments of the last 2 hours suggest that
>> "java -listmods" does not show user modules in its output, only JDK
>> modules:
>>java -mp jmods -listmods
>
> -listmods will list the names of the modules in the application's module
> graph. In this case, there isn't any initial module (application) specified
> so it defaults to the system modules. I expect this will do what you want:
>
> java -mp jmods -addmods org.joda.convert -listmods

So, after about an hour yesterday, I did try that and it still didn't
work. That must have been because jmod doesn't work fully. I've tried
it now, and with -addmods it does find the modular jar file.

However, my feedback is that the -listmods option is confusing. An
option called "list" is something I expect to list all things, not to
be clever and only show some subset. At the very least, I consider it
vital for there to be a tool (preferably java.exe) that can list *all*
available modules given a module path. Maybe the current -listmods
should be renamed to -applicationmods or -initialmods, or split to
-listmods:app and -listmods:all ?  (Being able to list everything
available feels like module 101).

Note to Alex Buckley's response, I haven't used the link or image
levels yet. The feedback above applies without using those tools.

>> My experiments also only worked with modular jar files. The jmod
>> generated files were not recognized as modules at all.
>
> There are a few open issues around the JMOD format. At this time (and stated
> in JEP 261), JMOD files can only be used at compile-time or link-time. So
> for now at least, they are ignored at run-time.

Bear in mind that the help/usage text of the jar.exe tool has not been
updated. As such, jmod.exe looks like its the only thing capable of
working with modules.

Stephen

Re: Jigsaw @ JavaOne 2015

[Stephen Colebourne]

I've also been somewhat concerned that the two namespaces are similar,
but concluded that it is best that way. Using underscores, dashes or
anything else would be more confusing, given that the emphasis is on a
namespace concept within the JLS.

I do find remembering which is which, module vs package, in
module-info.java is a little confusing, but I suspect we'll get used
to it. I wouldn't object to using "requires module com.foo.bar"
however.

I'm more concerned about some other aspects of the module spec, which
I'm still pondering.

Stephen


On 28 October 2015 at 16:59, David M. Lloyd  wrote:
> We've been (with JBoss Modules and thus our various application server
> offerings) using module name conventions that match package names for
> several years, and the number of people who have actually been confused by
> it to my knowledge is exactly zero.
>
> The actual problem is probably quite overstated.  People just don't seem to
> have trouble with this (nor do people generally seem to get confused by, for
> example, a C++ library name being the same as the root C++ namespace used by
> that library, to draw another language equivalent).
>
>
> On 10/28/2015 06:56 AM, Remi Forax wrote:
>>
>> Hi Marrio,
>>
>> When creating a new application, using the prefix of the packages as name
>> for a module seems intuitive and using '_' instead of '.' as separator
>> inside the module name avoid the unecessary confusion for a human between a
>> package and a module with the same name, it's just a code convention.
>>
>> When retrofitting an old application, like by example the JDK, you will
>> group packages that have no a common prefix name or the common prefix can be
>> used for several modules, in that case, having a module named java.base but
>> no package java.base.something seems counter intuitive, using '_' instead of
>> '.' make clear that a module name is just a name.
>>
>> regards,
>> Rémi
>>
>> - Mail original -
>>>
>>> De: "Mario Torre" 
>>> À: "Paul Benedict" 
>>> Cc: jigsaw-dev@openjdk.java.net
>>> Envoyé: Mardi 27 Octobre 2015 23:41:05
>>> Objet: Re: Jigsaw @ JavaOne 2015
>>>
>>> 2015-10-27 22:13 GMT+01:00 Paul Benedict :

 Thanks Mark. Great slides. I'd just like to throw out my impression
 (again)
 that module names with dots look like packages. How receptive is the EG
 to
 changing it to underscores?
>>>
>>>
>>> I think that this is the exact point, mapping to package seems quite
>>> intuitive as it represents directly the content of the module.
>>>
>>> Cheers,
>>> Mario
>>>
>>> --
>>> pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
>>> Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF
>>>
>>> Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
>>> Proud GNU Classpath developer: http://www.classpath.org/
>>> OpenJDK: http://openjdk.java.net/projects/caciocavallo/
>>>
>>> Please, support open standards:
>>> http://endsoftpatents.org/
>>>
>
> --
> - DML

Compilation feedback and version question

[Stephen Colebourne]

I can confirm that the current EA build
1.9.0-ea-jigsaw-nightly-h3660-20151022-b86 allowed a successful build
using maven (with tests) of the OpenGamma test case, which is a step
forward from when I last tried it.

When running "java -listmods" I get outpuy including this:

java.base@9.0
java.compact1@9.0
java.compact2@9.0
java.compact3@9.0
java.compiler@9.0
java.corba@9.0

But when I look at the JDK source module-info.java file:
http://hg.openjdk.java.net/jigsaw/jake/jdk/file/1b632228f7e5/src/java.base/share/classes/module-info.java
It does not contain the version number "9.0".

Where is the version coming from? It is weird to see the version
number so closely associated with the module in -listmods when so much
of the mood music of the current Jigsaw work is that the version is
"not our problem guv".

Stephen