Re: modulepath and classpath mixture

[Jonathan Gibbons]

Rémi,

I think all of us working on Jigsaw have equally shared the experience of 
figuring out how to do testing in this modular world.

Currently, the solution to your problem is to use javac's -XaddReads option, 
which is to javac's compilation environment what the launcher's -XaddReads 
option is to the execution environment.

Just as the runtime provides a family of options to configure the modules is 
use at runtime,
-upgrademodulepath -modulepath -addmods -limitmods -XaddReads 
-XaddExports -Xpatch etc
so too does javac have an equivalent set of options to configure the modules in 
use at compile time.

The primary additions/differences at compile time:
-modulesourcepath   (obviously source in not applicable at runtime)
-system (when compiling, you can override the default 
target system)
-Xmodulefor compiling additional classes to patch into 
a module

-- Jon



On 04/01/2016 02:55 PM, fo...@univ-mlv.fr wrote:

Hi Jon,
thanks for taking the time to explain the whole picture, as you know, you are 
probably the one with most experience about testing in the modular world.

This threads starts with this question, how to do unit testing with Maven when 
everything is a module, your case 3, so i will focus on that case.
Here is my question, in case of 3a or 3b, i don't see how to explain to javac 
how to compile if there are some specific dependencies for the test (that are 
also modules).

regards,
Rémi

- Mail original -

De: "Jonathan Gibbons" <jonathan.gibb...@oracle.com>
À: "Remi Forax" <fo...@univ-mlv.fr>, "Alan Bateman" <alan.bate...@oracle.com>
Cc: "Russell Gold" <russell.g...@oracle.com>, "jigsaw-dev" 
<jigsaw-dev@openjdk.java.net>
Envoyé: Jeudi 31 Mars 2016 03:29:55
Objet: Re: modulepath and classpath mixture

I have been following this thread, and it strongly reminds me of Joe
Darcy's parables of elephants and blind men. [1,2]

In this context, the discussion has been about testing, and the
underlying presumption that one size fits all.

I venture to suggest that one size does /not/ fit all, and that we have
to be able to support a wide range of testing paradigms.

1) Some folk will want to do black box testing, with their tests
contained in their own first class module, to exercise the code the way
that real clients might do

2) Some folk will want to do black box testing, with their test code on
the classpath, in the unnamed module, because they don't want to update
their test code

3) Some folk will want to do white box testing, using code conventions
is widespread use today, where the test code is alongside the code being
tested, in the same package.   This generates two sub-cases:   a) is the
test code compiled into the module itself, in a special test-specific
build, or  b) is the test code dynamically patched into the module

4) Some folk will want to do white box testing, but will be less
concerned about the package used to contain the package.  That has the
same two sub-cases:a) is the test code compiled into the module
itself, in a special test-specific build, or  b) is the test code
dynamically patched into the module

Now you take all those scenarios, and cross-product them with issues
like, is there a test framework involved, such as JUnit or TestNG, and,
where should that framework be placed: can it be an automatic module, or
it is sufficient to put the framework on the classpath, in the unnamed
module?

When you understand all that, then you begin to see the shape of the
elephant in the room that is the testing problem.

But, to stretch the analogy to breaking point,  it's not the only
elephant in the room.  There's another, different, elephant called
"strong encapsulation", and there is a strong conflict between the
desire for easy white box testing and the desire for strong
encapsulation. While the past 20 years of Java have led to easy simple
white box testing, leveraging split packages by adding jars onto the
class path, we've also seen the the problems that such techniques can
lead to ... hence the desire for Project Jigsaw, and the requirement [3]
for strong encapsulation.  We have to accept that the better we succeed
at strong encapsulation, the harder it will be to use the old ways of
white box testing. Conversely, the more we hold to the old ways of
working, the less we will succeed at satisfying the goal of strong
encapsulation.

So, ultimately, the trick is to figure out how to walk the tightrope
between the two elephants in the room.  Previously, as a community,
we've built tools to help manage the task. Now, it is time for the tools
and our practices to evolve, to meet the new demands of testing in a
modular world.

Do we have all the answers today? Almost certainly not. But I will note
one of the unheralded success stories of Jigsaw and OpenJDK. Testing has
always been important, and w

Re: modulepath and classpath mixture

[forax]

Hi Jon,
thanks for taking the time to explain the whole picture, as you know, you are 
probably the one with most experience about testing in the modular world.

This threads starts with this question, how to do unit testing with Maven when 
everything is a module, your case 3, so i will focus on that case.
Here is my question, in case of 3a or 3b, i don't see how to explain to javac 
how to compile if there are some specific dependencies for the test (that are 
also modules).

regards,
Rémi

- Mail original -
> De: "Jonathan Gibbons" <jonathan.gibb...@oracle.com>
> À: "Remi Forax" <fo...@univ-mlv.fr>, "Alan Bateman" <alan.bate...@oracle.com>
> Cc: "Russell Gold" <russell.g...@oracle.com>, "jigsaw-dev" 
> <jigsaw-dev@openjdk.java.net>
> Envoyé: Jeudi 31 Mars 2016 03:29:55
> Objet: Re: modulepath and classpath mixture
> 
> I have been following this thread, and it strongly reminds me of Joe
> Darcy's parables of elephants and blind men. [1,2]
> 
> In this context, the discussion has been about testing, and the
> underlying presumption that one size fits all.
> 
> I venture to suggest that one size does /not/ fit all, and that we have
> to be able to support a wide range of testing paradigms.
> 
> 1) Some folk will want to do black box testing, with their tests
> contained in their own first class module, to exercise the code the way
> that real clients might do
> 
> 2) Some folk will want to do black box testing, with their test code on
> the classpath, in the unnamed module, because they don't want to update
> their test code
> 
> 3) Some folk will want to do white box testing, using code conventions
> is widespread use today, where the test code is alongside the code being
> tested, in the same package.   This generates two sub-cases:   a) is the
> test code compiled into the module itself, in a special test-specific
> build, or  b) is the test code dynamically patched into the module
> 
> 4) Some folk will want to do white box testing, but will be less
> concerned about the package used to contain the package.  That has the
> same two sub-cases:a) is the test code compiled into the module
> itself, in a special test-specific build, or  b) is the test code
> dynamically patched into the module
> 
> Now you take all those scenarios, and cross-product them with issues
> like, is there a test framework involved, such as JUnit or TestNG, and,
> where should that framework be placed: can it be an automatic module, or
> it is sufficient to put the framework on the classpath, in the unnamed
> module?
> 
> When you understand all that, then you begin to see the shape of the
> elephant in the room that is the testing problem.
> 
> But, to stretch the analogy to breaking point,  it's not the only
> elephant in the room.  There's another, different, elephant called
> "strong encapsulation", and there is a strong conflict between the
> desire for easy white box testing and the desire for strong
> encapsulation. While the past 20 years of Java have led to easy simple
> white box testing, leveraging split packages by adding jars onto the
> class path, we've also seen the the problems that such techniques can
> lead to ... hence the desire for Project Jigsaw, and the requirement [3]
> for strong encapsulation.  We have to accept that the better we succeed
> at strong encapsulation, the harder it will be to use the old ways of
> white box testing. Conversely, the more we hold to the old ways of
> working, the less we will succeed at satisfying the goal of strong
> encapsulation.
> 
> So, ultimately, the trick is to figure out how to walk the tightrope
> between the two elephants in the room.  Previously, as a community,
> we've built tools to help manage the task. Now, it is time for the tools
> and our practices to evolve, to meet the new demands of testing in a
> modular world.
> 
> Do we have all the answers today? Almost certainly not. But I will note
> one of the unheralded success stories of Jigsaw and OpenJDK. Testing has
> always been important, and we have managed to keep running almost all of
> the JDK unit and regression tests with little to no change in
> functionality in almost all of the tests, and we have started supporting
> new testing scenarios as well. Looking back at the list I gave at the
> beginning, support for 1 is coming available, but not yet widely used,
> many JDK tests use case 2,  JDK has a number of tests that use 3b and
> 4b, and we have tests that use JUnit and TestNG.
> 
> So, command line options like -XaddExports, -Xpatch, -Xmodule, etc, may
> not be pretty, but they can be composed to cover a wide range of testing
> scenarios, without giving up too much on the goal of strong
> encapsulation, and that at least is some degree of success.
> 
> -- Jon
> 
> [1]
> https://blogs.oracle.com/darcy/entry/project_coin_elephants_knapsacks_language
> [2] http://en.wikipedia.org/wiki/Blind_Men_and_an_Elephant
> [3] http://openjdk.java.net/projects/jigsaw/spec/reqs/
> 

Re: modulepath and classpath mixture

[Russell Gold]

that delivering tests as a separated module is a bad idea.
>>> >>
>>> >> I see that from the point of a developer, seeing the code and the test 
>>> >> as different modules can be attractive because everything seems to be 
>>> >> put at the right place but there is a big drawback. Because modules are 
>>> >> reified at runtime, it means that the runtime environment of the tests 
>>> >> will be different from the production environment.
>>> >
>>> > This last sentence doesn't make sense to me - tests are not run in a
>>> > production environment.
>>> >
>>> > Tests have all the qualities of modules - code, dependencies,
>>> > compilation phase, deployment. The only special part is the need for
>>> > special "friend-like" access, which Jigsaw already has for other cases
>>> > (the export...to clause).
>>> >
>>> > Put simply, I consider that module =
>>> > deployment-artifact-with-dependencies. With that mental model, putting
>>> > tests inside the module is just not acceptable, because tests should
>>> > not be deployed with the main application and they have different
>>> > dependencies. If we disagree that module =
>>> > deployment-artifact-with-dependencies, then perhaps we have bigger
>>> > problems to solve here.
>>> >
>>> > Stephen
>>> > (And to Paul Benedict, the classpath is going to die over time, so any
>>> > solution that uses that is flawed IMO).
>>> >
>>> >
>>> >> So as Alan said, from the jigsaw point of view at runtime, the tests and 
>>> >> the code should be in the same module.
>>> >>
>>> >> So the building tools have to come with a way to support to have 2 
>>> >> different module-info.java in two different folders and package them as 
>>> >> one module,
>>> >> maybe javac should help by providing a way to merge 2 module-info at 
>>> >> compile time.
>>> >>
>>> >> Rémi
>>> >>
>>> >> - Mail original -
>>> >>> De: "Alan Bateman" <alan.bate...@oracle.com 
>>> >>> <mailto:alan.bate...@oracle.com>>
>>> >>> À: "Stephen Colebourne" <scolebou...@joda.org 
>>> >>> <mailto:scolebou...@joda.org>>, "jigsaw-dev" 
>>> >>> <jigsaw-dev@openjdk.java.net <mailto:jigsaw-dev@openjdk.java.net>>
>>> >>> Envoyé: Mercredi 23 Mars 2016 16:18:50
>>> >>> Objet: Re: modulepath and classpath mixture
>>> >>>
>>> >>>
>>> >>> On 23/03/2016 14:42, Stephen Colebourne wrote:
>>> >>>> :
>>> >>>>
>>> >>>> I don't particularly care what the mechanism is for this, but at the
>>> >>>> requirements level:
>>> >>>> - there are two modules - main and test
>>> >>>> - each has its own source tree
>>> >>>> - each has its own dependencies
>>> >>>> - each is released separately
>>> >>>> - each could be hosted on a central repo
>>> >>>> - the test module needs to be able to contain the same packages as the
>>> >>>> main module
>>> >>>> - the test module needs to be able to invoke package-scoped code in
>>> >>>> the same package in the main module
>>> >>>>
>>> >>>> To clarify further consider 4 modules, A, B, A-test and B-test where B
>>> >>>> depends on A. Module A-test may have a method foo() that uses package
>>> >>>> scope to access something in A. Module B-test will depend on A-test
>>> >>>> and rely on foo() to get access to that internal object.
>>> >>> To your list, I would add the ability to make use of testing frameworks
>>> >>> like TestNG and JUnit.
>>> >>>
>>> >>> In any case, and as things currently stand, you've got most of the
>>> >>> above. One differences is that the tests are not a separate module, they
>>> >>> are instead compiled and run in a way that patches the main module. The
>>> >>> second difference is that they don't have their own module declaration,
>>> >>> instead the compilation or run augments the dependences with any
>>> >>> additional dependences that the tests have. As I said, if they tools
>>> >>> makes it easy then I don't think it's too bad.
>>> >>>
>>> >>>>
>>> >>>> (Note that I view the thread discussion of
>>> >>>> references to test classes on the classpath as another hack.
>>> >>>>
>>> >>> Packages can't be split between modules and classpath so there is no
>>> >>> support for that.
>>> >>>
>>> >>> -Alan
>>> >>>

Re: modulepath and classpath mixture

[Jonathan Gibbons]

On 03/30/2016 02:45 PM, fo...@univ-mlv.fr wrote:





*De: *"Ali Ebrahimi" <ali.ebrahimi1...@gmail.com>
*À: *"Remi Forax" <fo...@univ-mlv.fr>
*Cc: *"Alan Bateman" <alan.bate...@oracle.com>, "Jonathan Gibbons"
<jonathan.gibb...@oracle.com>, "jigsaw-dev"
<jigsaw-dev@openjdk.java.net>
*Envoyé: *Mercredi 30 Mars 2016 17:12:22
*Objet: *Re: modulepath and classpath mixture

So, do you suggest partial modules or module fragments?


A kind of exploded module with sources available in more than one 
directory.
Classes are still in one modular jar with one module-info.class (so at 
runtime, there is only one module).



Why we make things so complex for writing single test method. I
think testing is an essential part of development, so modular java
should have first class support for that.
I don't see command line options as developer friendly solution,
even things gets worse when have dozen of modules.


That's another question, i think we first need to agree that we just 
want to have the main code and the test code into different directory 
and then we can see if javac -Xmodule is the best syntax or if by 
example -modulesourcepath can accept to have several directories that 
describe the same module.


-modulesourcepath can accept several source directories that together 
comprise the source for a single module.  If nothing else, you can see 
this utilized in the JDK build, where the source structure for a module 
may contain subdirectories for platform-independent classes, 
platform-specific classes, and dynamically generated classes.


The primary constraints are that there is only one module-info.java 
file, and the mapping to a class output directory.


There is nothing to prevent you recompiling an entire module, including 
test code, to create a new version of the module, including the test 
code. If you so choose.


-- Jon



cheers,
Rémi



On Wed, Mar 30, 2016 at 6:42 PM, Remi Forax <fo...@univ-mlv.fr
<mailto:fo...@univ-mlv.fr>> wrote:

Alan, Jon,
i think javac -Xmodule should merge the module-info.java from
the existing module and the one declared in the directory,
with the current semantics of the module-info, merging of
modules is easy and with no corner cases,
so for testing, the test will be able to declare their own
dependencies inside their own module-info.java.

Proposed semantics for merging,
- do the union of the required modules
  - if one required module is required publicly, it will be
required publicly.
- do the union of the exported packages
  - if one exported package is restricted, do the union of the
restriction
- do the union of the uses.
- do the union of the provides.

so merging two modules is symmetric and will always succeed.

Rémi

- Mail original -
> De: "Alan Bateman" <alan.bate...@oracle.com
<mailto:alan.bate...@oracle.com>>
> À: "Russell Gold" <russell.g...@oracle.com
<mailto:russell.g...@oracle.com>>
> Cc: "jigsaw-dev" <jigsaw-dev@openjdk.java.net
<mailto:jigsaw-dev@openjdk.java.net>>
> Envoyé: Mercredi 30 Mars 2016 15:45:03
> Objet: Re: modulepath and classpath mixture
>
> On 30/03/2016 13:28, Russell Gold wrote:
> > :
> >
> >
> > So if the tests and main code are both in directories,
which they have been
> > up to now in Maven, why would there be a problem? Both
would be in the
> > unnamed module and able to access one another.
> >
> There shouldn't any issue there, it should just work as it
has always done.
>
> The thread here has meandered a bit but I think the scenario
under
> discussion is tests for a module that need to nestmate with
the module
> under test. The tests are in their own test tree. The tests
are compiled
> separately from the module they test and may have additional
dependences
> (such as on TestNG or JUnit for example). When compiling or
running then
> the tests need to access public types in non-exported
packages and maybe
> package private members too. The support for this has been
in jake for a
> long time but involves command line options that many
developers or
> build environments won't immediately grok. In particular the
tests have
> to be compiled "as if" they augm

Re: modulepath and classpath mixture

[forax]

- Mail original -

> De: "Ali Ebrahimi" <ali.ebrahimi1...@gmail.com>
> À: "Remi Forax" <fo...@univ-mlv.fr>
> Cc: "Alan Bateman" <alan.bate...@oracle.com>, "Jonathan Gibbons"
> <jonathan.gibb...@oracle.com>, "jigsaw-dev" <jigsaw-dev@openjdk.java.net>
> Envoyé: Mercredi 30 Mars 2016 17:12:22
> Objet: Re: modulepath and classpath mixture

> So, do you suggest partial modules or module fragments?

A kind of exploded module with sources available in more than one directory. 
Classes are still in one modular jar with one module-info.class (so at runtime, 
there is only one module). 

> Why we make things so complex for writing single test method. I think testing
> is an essential part of development, so modular java should have first class
> support for that.
> I don't see command line options as developer friendly solution, even things
> gets worse when have dozen of modules.

That's another question, i think we first need to agree that we just want to 
have the main code and the test code into different directory and then we can 
see if javac -Xmodule is the best syntax or if by example -modulesourcepath can 
accept to have several directories that describe the same module. 

cheers, 
Rémi 

> On Wed, Mar 30, 2016 at 6:42 PM, Remi Forax < fo...@univ-mlv.fr > wrote:

> > Alan, Jon,
> 
> > i think javac -Xmodule should merge the module-info.java from the existing
> > module and the one declared in the directory,
> 
> > with the current semantics of the module-info, merging of modules is easy
> > and
> > with no corner cases,
> 
> > so for testing, the test will be able to declare their own dependencies
> > inside their own module-info.java.
> 

> > Proposed semantics for merging,
> 
> > - do the union of the required modules
> 
> > - if one required module is required publicly, it will be required
> > publicly.
> 
> > - do the union of the exported packages
> 
> > - if one exported package is restricted, do the union of the restriction
> 
> > - do the union of the uses.
> 
> > - do the union of the provides.
> 

> > so merging two modules is symmetric and will always succeed.
> 

> > Rémi
> 

> > - Mail original -
> 
> > > De: "Alan Bateman" < alan.bate...@oracle.com >
> 
> > > À: "Russell Gold" < russell.g...@oracle.com >
> 
> > > Cc: "jigsaw-dev" < jigsaw-dev@openjdk.java.net >
> 
> > > Envoyé: Mercredi 30 Mars 2016 15:45:03
> 
> > > Objet: Re: modulepath and classpath mixture
> 
> > >
> 
> > > On 30/03/2016 13:28, Russell Gold wrote:
> 
> > > > :
> 
> > > >
> 
> > > >
> 
> > > > So if the tests and main code are both in directories, which they have
> > > > been
> 
> > > > up to now in Maven, why would there be a problem? Both would be in the
> 
> > > > unnamed module and able to access one another.
> 
> > > >
> 
> > > There shouldn't any issue there, it should just work as it has always
> > > done.
> 
> > >
> 
> > > The thread here has meandered a bit but I think the scenario under
> 
> > > discussion is tests for a module that need to nestmate with the module
> 
> > > under test. The tests are in their own test tree. The tests are compiled
> 
> > > separately from the module they test and may have additional dependences
> 
> > > (such as on TestNG or JUnit for example). When compiling or running then
> 
> > > the tests need to access public types in non-exported packages and maybe
> 
> > > package private members too. The support for this has been in jake for a
> 
> > > long time but involves command line options that many developers or
> 
> > > build environments won't immediately grok. In particular the tests have
> 
> > > to be compiled "as if" they augment the already compiled module - that
> 
> > > is what javac -Xmodule is about. There is no need to co-locate source
> 
> > > files or class files of course. When run then the -Xpatch option is what
> 
> > > brings the tests and the module classes together. If we get the tools
> 
> > > right then most developers won't ever see this of course.
> 
> > >
> 
> > > One other thing to say that we've already been through some of this with
> 
> > > the JDK tests. The jtreg test harness that we use for the JDK tests has
> 
> > > been updated (thanks to Jon Gibbons) with us

Re: modulepath and classpath mixture

[Paul Benedict]

pendencies. With that mental model, putting
>>> > tests inside the module is just not acceptable, because tests should
>>> > not be deployed with the main application and they have different
>>> > dependencies. If we disagree that module =
>>> > deployment-artifact-with-dependencies, then perhaps we have bigger
>>> > problems to solve here.
>>> >
>>> > Stephen
>>> > (And to Paul Benedict, the classpath is going to die over time, so any
>>> > solution that uses that is flawed IMO).
>>> >
>>> >
>>> >> So as Alan said, from the jigsaw point of view at runtime, the tests
>>> and the code should be in the same module.
>>> >>
>>> >> So the building tools have to come with a way to support to have 2
>>> different module-info.java in two different folders and package them as one
>>> module,
>>> >> maybe javac should help by providing a way to merge 2 module-info at
>>> compile time.
>>> >>
>>> >> Rémi
>>> >>
>>> >> - Mail original -
>>> >>> De: "Alan Bateman" <alan.bate...@oracle.com>
>>> >>> À: "Stephen Colebourne" <scolebou...@joda.org>, "jigsaw-dev" <
>>> jigsaw-dev@openjdk.java.net>
>>> >>> Envoyé: Mercredi 23 Mars 2016 16:18:50
>>> >>> Objet: Re: modulepath and classpath mixture
>>> >>>
>>> >>>
>>> >>> On 23/03/2016 14:42, Stephen Colebourne wrote:
>>> >>>> :
>>> >>>>
>>> >>>> I don't particularly care what the mechanism is for this, but at the
>>> >>>> requirements level:
>>> >>>> - there are two modules - main and test
>>> >>>> - each has its own source tree
>>> >>>> - each has its own dependencies
>>> >>>> - each is released separately
>>> >>>> - each could be hosted on a central repo
>>> >>>> - the test module needs to be able to contain the same packages as
>>> the
>>> >>>> main module
>>> >>>> - the test module needs to be able to invoke package-scoped code in
>>> >>>> the same package in the main module
>>> >>>>
>>> >>>> To clarify further consider 4 modules, A, B, A-test and B-test
>>> where B
>>> >>>> depends on A. Module A-test may have a method foo() that uses
>>> package
>>> >>>> scope to access something in A. Module B-test will depend on A-test
>>> >>>> and rely on foo() to get access to that internal object.
>>> >>> To your list, I would add the ability to make use of testing
>>> frameworks
>>> >>> like TestNG and JUnit.
>>> >>>
>>> >>> In any case, and as things currently stand, you've got most of the
>>> >>> above. One differences is that the tests are not a separate module,
>>> they
>>> >>> are instead compiled and run in a way that patches the main module.
>>> The
>>> >>> second difference is that they don't have their own module
>>> declaration,
>>> >>> instead the compilation or run augments the dependences with any
>>> >>> additional dependences that the tests have. As I said, if they tools
>>> >>> makes it easy then I don't think it's too bad.
>>> >>>
>>> >>>>
>>> >>>> (Note that I view the thread discussion of
>>> >>>> references to test classes on the classpath as another hack.
>>> >>>>
>>> >>> Packages can't be split between modules and classpath so there is no
>>> >>> support for that.
>>> >>>
>>> >>> -Alan
>>> >>>
>>>
>>
>

Re: modulepath and classpath mixture

[Russell Gold]

; (And to Paul Benedict, the classpath is going to die over time, so any
>> > solution that uses that is flawed IMO).
>> >
>> >
>> >> So as Alan said, from the jigsaw point of view at runtime, the tests and 
>> >> the code should be in the same module.
>> >>
>> >> So the building tools have to come with a way to support to have 2 
>> >> different module-info.java in two different folders and package them as 
>> >> one module,
>> >> maybe javac should help by providing a way to merge 2 module-info at 
>> >> compile time.
>> >>
>> >> Rémi
>> >>
>> >> - Mail original -
>> >>> De: "Alan Bateman" <alan.bate...@oracle.com 
>> >>> <mailto:alan.bate...@oracle.com>>
>> >>> À: "Stephen Colebourne" <scolebou...@joda.org 
>> >>> <mailto:scolebou...@joda.org>>, "jigsaw-dev" 
>> >>> <jigsaw-dev@openjdk.java.net <mailto:jigsaw-dev@openjdk.java.net>>
>> >>> Envoyé: Mercredi 23 Mars 2016 16:18:50
>> >>> Objet: Re: modulepath and classpath mixture
>> >>>
>> >>>
>> >>> On 23/03/2016 14:42, Stephen Colebourne wrote:
>> >>>> :
>> >>>>
>> >>>> I don't particularly care what the mechanism is for this, but at the
>> >>>> requirements level:
>> >>>> - there are two modules - main and test
>> >>>> - each has its own source tree
>> >>>> - each has its own dependencies
>> >>>> - each is released separately
>> >>>> - each could be hosted on a central repo
>> >>>> - the test module needs to be able to contain the same packages as the
>> >>>> main module
>> >>>> - the test module needs to be able to invoke package-scoped code in
>> >>>> the same package in the main module
>> >>>>
>> >>>> To clarify further consider 4 modules, A, B, A-test and B-test where B
>> >>>> depends on A. Module A-test may have a method foo() that uses package
>> >>>> scope to access something in A. Module B-test will depend on A-test
>> >>>> and rely on foo() to get access to that internal object.
>> >>> To your list, I would add the ability to make use of testing frameworks
>> >>> like TestNG and JUnit.
>> >>>
>> >>> In any case, and as things currently stand, you've got most of the
>> >>> above. One differences is that the tests are not a separate module, they
>> >>> are instead compiled and run in a way that patches the main module. The
>> >>> second difference is that they don't have their own module declaration,
>> >>> instead the compilation or run augments the dependences with any
>> >>> additional dependences that the tests have. As I said, if they tools
>> >>> makes it easy then I don't think it's too bad.
>> >>>
>> >>>>
>> >>>> (Note that I view the thread discussion of
>> >>>> references to test classes on the classpath as another hack.
>> >>>>
>> >>> Packages can't be split between modules and classpath so there is no
>> >>> support for that.
>> >>>
>> >>> -Alan
>> >>>

Re: modulepath and classpath mixture

[Ali Ebrahimi]

So, do you suggest partial modules or module fragments?

Why we make things so complex for writing single test method. I think
testing is an essential part of development, so modular java should have
first class support for that.
I don't see command line options as developer friendly solution, even
things gets worse when have dozen of modules.

On Wed, Mar 30, 2016 at 6:42 PM, Remi Forax <fo...@univ-mlv.fr> wrote:

> Alan, Jon,
> i think javac -Xmodule should merge the module-info.java from the existing
> module and the one declared in the directory,
> with the current semantics of the module-info, merging of modules is easy
> and with no corner cases,
> so for testing, the test will be able to declare their own dependencies
> inside their own module-info.java.
>
> Proposed semantics for merging,
> - do the union of the required modules
>   - if one required module is required publicly, it will be required
> publicly.
> - do the union of the exported packages
>   - if one exported package is restricted, do the union of the restriction
> - do the union of the uses.
> - do the union of the provides.
>
> so merging two modules is symmetric and will always succeed.
>
> Rémi
>
> - Mail original -
> > De: "Alan Bateman" <alan.bate...@oracle.com>
> > À: "Russell Gold" <russell.g...@oracle.com>
> > Cc: "jigsaw-dev" <jigsaw-dev@openjdk.java.net>
> > Envoyé: Mercredi 30 Mars 2016 15:45:03
> > Objet: Re: modulepath and classpath mixture
> >
> > On 30/03/2016 13:28, Russell Gold wrote:
> > > :
> > >
> > >
> > > So if the tests and main code are both in directories, which they have
> been
> > > up to now in Maven, why would there be a problem? Both would be in the
> > > unnamed module and able to access one another.
> > >
> > There shouldn't any issue there, it should just work as it has always
> done.
> >
> > The thread here has meandered a bit but I think the scenario under
> > discussion is tests for a module that need to nestmate with the module
> > under test. The tests are in their own test tree. The tests are compiled
> > separately from the module they test and may have additional dependences
> > (such as on TestNG or JUnit for example). When compiling or running then
> > the tests need to access public types in non-exported packages and maybe
> > package private members too. The support for this has been in jake for a
> > long time but involves command line options that many developers or
> > build environments won't immediately grok. In particular the tests have
> > to be compiled "as if" they augment the already compiled module - that
> > is what javac -Xmodule is about. There is no need to co-locate source
> > files or class files of course. When run then the -Xpatch option is what
> > brings the tests and the module classes together. If we get the tools
> > right then most developers won't ever see this of course.
> >
> > One other thing to say that we've already been through some of this with
> > the JDK tests. The jtreg test harness that we use for the JDK tests has
> > been updated (thanks to Jon Gibbons) with useful support for modules
> > [1]. It's enough for us to write tests that use JDK-internal APIs or
> > write tests that nestmate with types in system modules so that they get
> > access to package private type or public types in non-exported packages.
> > It has rudimentary support for user modules too. Additional dependences
> > are still an issue but our tests don't require additional dependences
> > beyond TestNG. The test harness employs a bit of hackery to get things
> > done, important when starting out, but I expect will go away in time.
> >
> > -Alan.
> >
> > [1]
> >
> http://hg.openjdk.java.net/code-tools/jtreg/raw-file/tip/src/share/doc/javatest/regtest/tag-spec.html
> >
> >
> >
>



-- 

Best Regards,
Ali Ebrahimi

Re: modulepath and classpath mixture

[Paul Benedict]

Russell, if you have a module with package X and a classpath jar with
package X, the module can't see package X from the classpath.

In the last several posts, there's been discussion on putting tests on the
classpath; keeping the "main" code in the module. So given what I stated
above, that's what I've been referring to.

Cheers,
Paul

On Wed, Mar 30, 2016 at 7:28 AM, Russell Gold <russell.g...@oracle.com>
wrote:

> Hi Paul,
>
> Could you explain? What kind of code do you mean cannot access it? And
> what do you mean by “a split package situation”?
>
> From a conversation I had with Alan, earlier in this thread:
>
>
> On Mar 23, 2016, at 11:42 AM, Alan Bateman <alan.bate...@oracle.com>
> wrote:
>
> On 23/03/2016 14:15, Russell Gold wrote:
>
> Here are my assumptions, which you can correct.
>
> 1. A jar or classes directory placed on a class path are treated as part
> of the unnamed module
>
> Yes
>
>
> So if the tests and main code are both in directories, which they have
> been up to now in Maven, why would there be a problem? Both would be in the
> unnamed module and able to access one another.
>
> - Russ
>
> On Mar 29, 2016, at 10:47 PM, Paul Benedict <pbened...@apache.org> wrote:
>
> Russell, when you drop a jar on the classpath, module code will not be
> able to access it in a split package situation. That's the big barrier
> here. Maven test projects are typically written with the same package
> shared with the "main" code.
>
> Cheers,
> Paul
>
> On Tue, Mar 29, 2016 at 11:33 AM, Russell Gold <russell.g...@oracle.com>
> wrote:
>
>> Hi Stephen,
>>
>> Why do you need any kind of friend access?
>>
>> It seems to me that this is making things harder than they need to be.
>> The tests can simply run with the production code on the class path, and
>> then there are no module issues at all.
>>
>> I think a larger problem is that you can do what I just said with the
>> jars, even a jar which has been designated as a module by virtue of having
>> a module-info.class in it. That means that, when users are up taking jars,
>> they are not prevented from accessing module internals. They can put the
>> jars on the module path, of course, but they can still use them on the
>> class path!
>>
>> - Russ
>>
>> >
>> > On 28 March 2016 at 11:13, Remi Forax <fo...@univ-mlv.fr> wrote:
>> >> Hi Stephen, Hi all,
>> >> I think that delivering tests as a separated module is a bad idea.
>> >>
>> >> I see that from the point of a developer, seeing the code and the test
>> as different modules can be attractive because everything seems to be put
>> at the right place but there is a big drawback. Because modules are reified
>> at runtime, it means that the runtime environment of the tests will be
>> different from the production environment.
>> >
>> > This last sentence doesn't make sense to me - tests are not run in a
>> > production environment.
>> >
>> > Tests have all the qualities of modules - code, dependencies,
>> > compilation phase, deployment. The only special part is the need for
>> > special "friend-like" access, which Jigsaw already has for other cases
>> > (the export...to clause).
>> >
>> > Put simply, I consider that module =
>> > deployment-artifact-with-dependencies. With that mental model, putting
>> > tests inside the module is just not acceptable, because tests should
>> > not be deployed with the main application and they have different
>> > dependencies. If we disagree that module =
>> > deployment-artifact-with-dependencies, then perhaps we have bigger
>> > problems to solve here.
>> >
>> > Stephen
>> > (And to Paul Benedict, the classpath is going to die over time, so any
>> > solution that uses that is flawed IMO).
>> >
>> >
>> >> So as Alan said, from the jigsaw point of view at runtime, the tests
>> and the code should be in the same module.
>> >>
>> >> So the building tools have to come with a way to support to have 2
>> different module-info.java in two different folders and package them as one
>> module,
>> >> maybe javac should help by providing a way to merge 2 module-info at
>> compile time.
>> >>
>> >> Rémi
>> >>
>> >> - Mail original -
>> >>> De: "Alan Bateman" <alan.bate...@oracle.com>
>> >>> À: "Stephen Colebourne" <scolebou...@joda.org>,

Re: modulepath and classpath mixture

[Remi Forax]

Alan, Jon,
i think javac -Xmodule should merge the module-info.java from the existing 
module and the one declared in the directory,
with the current semantics of the module-info, merging of modules is easy and 
with no corner cases,
so for testing, the test will be able to declare their own dependencies inside 
their own module-info.java.

Proposed semantics for merging,
- do the union of the required modules
  - if one required module is required publicly, it will be required publicly. 
- do the union of the exported packages
  - if one exported package is restricted, do the union of the restriction
- do the union of the uses.
- do the union of the provides.

so merging two modules is symmetric and will always succeed.

Rémi

- Mail original -
> De: "Alan Bateman" <alan.bate...@oracle.com>
> À: "Russell Gold" <russell.g...@oracle.com>
> Cc: "jigsaw-dev" <jigsaw-dev@openjdk.java.net>
> Envoyé: Mercredi 30 Mars 2016 15:45:03
> Objet: Re: modulepath and classpath mixture
> 
> On 30/03/2016 13:28, Russell Gold wrote:
> > :
> >
> >
> > So if the tests and main code are both in directories, which they have been
> > up to now in Maven, why would there be a problem? Both would be in the
> > unnamed module and able to access one another.
> >
> There shouldn't any issue there, it should just work as it has always done.
> 
> The thread here has meandered a bit but I think the scenario under
> discussion is tests for a module that need to nestmate with the module
> under test. The tests are in their own test tree. The tests are compiled
> separately from the module they test and may have additional dependences
> (such as on TestNG or JUnit for example). When compiling or running then
> the tests need to access public types in non-exported packages and maybe
> package private members too. The support for this has been in jake for a
> long time but involves command line options that many developers or
> build environments won't immediately grok. In particular the tests have
> to be compiled "as if" they augment the already compiled module - that
> is what javac -Xmodule is about. There is no need to co-locate source
> files or class files of course. When run then the -Xpatch option is what
> brings the tests and the module classes together. If we get the tools
> right then most developers won't ever see this of course.
> 
> One other thing to say that we've already been through some of this with
> the JDK tests. The jtreg test harness that we use for the JDK tests has
> been updated (thanks to Jon Gibbons) with useful support for modules
> [1]. It's enough for us to write tests that use JDK-internal APIs or
> write tests that nestmate with types in system modules so that they get
> access to package private type or public types in non-exported packages.
> It has rudimentary support for user modules too. Additional dependences
> are still an issue but our tests don't require additional dependences
> beyond TestNG. The test harness employs a bit of hackery to get things
> done, important when starting out, but I expect will go away in time.
> 
> -Alan.
> 
> [1]
> http://hg.openjdk.java.net/code-tools/jtreg/raw-file/tip/src/share/doc/javatest/regtest/tag-spec.html
> 
> 
> 

Re: modulepath and classpath mixture

[Stephen Colebourne]

On 30 March 2016 at 14:45, Alan Bateman  wrote:
> The thread here has meandered a bit but I think the scenario under
> discussion is tests for a module that need to nestmate with the module under
> test. The tests are in their own test tree. The tests are compiled
> separately from the module they test and may have additional dependences
> (such as on TestNG or JUnit for example). When compiling or running then the
> tests need to access public types in non-exported packages and maybe package
> private members too.

Yes, plus there is a need for one set of  tests to depend on another
set of tests - ie. tests can be an artifact.

> The support for this has been in jake for a long time
> but involves command line options that many developers or build environments
> won't immediately grok. In particular the tests have to be compiled "as if"
> they augment the already compiled module - that is what javac -Xmodule is
> about. There is no need to co-locate source files or class files of course.
> When run then the -Xpatch option is what brings the tests and the module
> classes together. If we get the tools right then most developers won't ever
> see this of course.

This is perhaps a solution that works, but it comes across as highly
esoteric. Two different command line flags, augmenting/patching
modules. What I see is a nightmare to explain.

And while tools can make things easy, it is inadvisable for the design
for running something as basic as tests to be so complex that it
cannot practically be done without tools.

What I see as a better solution is the same approach as I argued for
with optional dependencies - enhancing the syntax of module-info.java
to avoid the need for command line flags

Stephen

Re: modulepath and classpath mixture

[Alan Bateman]

On 30/03/2016 13:28, Russell Gold wrote:

:


So if the tests and main code are both in directories, which they have been up 
to now in Maven, why would there be a problem? Both would be in the unnamed 
module and able to access one another.


There shouldn't any issue there, it should just work as it has always done.

The thread here has meandered a bit but I think the scenario under 
discussion is tests for a module that need to nestmate with the module 
under test. The tests are in their own test tree. The tests are compiled 
separately from the module they test and may have additional dependences 
(such as on TestNG or JUnit for example). When compiling or running then 
the tests need to access public types in non-exported packages and maybe 
package private members too. The support for this has been in jake for a 
long time but involves command line options that many developers or 
build environments won't immediately grok. In particular the tests have 
to be compiled "as if" they augment the already compiled module - that 
is what javac -Xmodule is about. There is no need to co-locate source 
files or class files of course. When run then the -Xpatch option is what 
brings the tests and the module classes together. If we get the tools 
right then most developers won't ever see this of course.


One other thing to say that we've already been through some of this with 
the JDK tests. The jtreg test harness that we use for the JDK tests has 
been updated (thanks to Jon Gibbons) with useful support for modules 
[1]. It's enough for us to write tests that use JDK-internal APIs or 
write tests that nestmate with types in system modules so that they get 
access to package private type or public types in non-exported packages. 
It has rudimentary support for user modules too. Additional dependences 
are still an issue but our tests don't require additional dependences 
beyond TestNG. The test harness employs a bit of hackery to get things 
done, important when starting out, but I expect will go away in time.


-Alan.

[1] 
http://hg.openjdk.java.net/code-tools/jtreg/raw-file/tip/src/share/doc/javatest/regtest/tag-spec.html

Re: modulepath and classpath mixture

[Russell Gold]

Hi Paul,

Could you explain? What kind of code do you mean cannot access it? And what do 
you mean by “a split package situation”?

From a conversation I had with Alan, earlier in this thread:


> On Mar 23, 2016, at 11:42 AM, Alan Bateman <alan.bate...@oracle.com> wrote:
> 
> On 23/03/2016 14:15, Russell Gold wrote:
>> Here are my assumptions, which you can correct.
>> 
>> 1. A jar or classes directory placed on a class path are treated as part of 
>> the unnamed module
> Yes


So if the tests and main code are both in directories, which they have been up 
to now in Maven, why would there be a problem? Both would be in the unnamed 
module and able to access one another.

- Russ

> On Mar 29, 2016, at 10:47 PM, Paul Benedict <pbened...@apache.org> wrote:
> 
> Russell, when you drop a jar on the classpath, module code will not be able 
> to access it in a split package situation. That's the big barrier here. Maven 
> test projects are typically written with the same package shared with the 
> "main" code. 
> 
> Cheers,
> Paul
> 
> On Tue, Mar 29, 2016 at 11:33 AM, Russell Gold <russell.g...@oracle.com 
> <mailto:russell.g...@oracle.com>> wrote:
> Hi Stephen,
> 
> Why do you need any kind of friend access?
> 
> It seems to me that this is making things harder than they need to be. The 
> tests can simply run with the production code on the class path, and then 
> there are no module issues at all.
> 
> I think a larger problem is that you can do what I just said with the jars, 
> even a jar which has been designated as a module by virtue of having a 
> module-info.class in it. That means that, when users are up taking jars, they 
> are not prevented from accessing module internals. They can put the jars on 
> the module path, of course, but they can still use them on the class path!
> 
> - Russ
> 
> >
> > On 28 March 2016 at 11:13, Remi Forax <fo...@univ-mlv.fr 
> > <mailto:fo...@univ-mlv.fr>> wrote:
> >> Hi Stephen, Hi all,
> >> I think that delivering tests as a separated module is a bad idea.
> >>
> >> I see that from the point of a developer, seeing the code and the test as 
> >> different modules can be attractive because everything seems to be put at 
> >> the right place but there is a big drawback. Because modules are reified 
> >> at runtime, it means that the runtime environment of the tests will be 
> >> different from the production environment.
> >
> > This last sentence doesn't make sense to me - tests are not run in a
> > production environment.
> >
> > Tests have all the qualities of modules - code, dependencies,
> > compilation phase, deployment. The only special part is the need for
> > special "friend-like" access, which Jigsaw already has for other cases
> > (the export...to clause).
> >
> > Put simply, I consider that module =
> > deployment-artifact-with-dependencies. With that mental model, putting
> > tests inside the module is just not acceptable, because tests should
> > not be deployed with the main application and they have different
> > dependencies. If we disagree that module =
> > deployment-artifact-with-dependencies, then perhaps we have bigger
> > problems to solve here.
> >
> > Stephen
> > (And to Paul Benedict, the classpath is going to die over time, so any
> > solution that uses that is flawed IMO).
> >
> >
> >> So as Alan said, from the jigsaw point of view at runtime, the tests and 
> >> the code should be in the same module.
> >>
> >> So the building tools have to come with a way to support to have 2 
> >> different module-info.java in two different folders and package them as 
> >> one module,
> >> maybe javac should help by providing a way to merge 2 module-info at 
> >> compile time.
> >>
> >> Rémi
> >>
> >> - Mail original -
> >>> De: "Alan Bateman" <alan.bate...@oracle.com 
> >>> <mailto:alan.bate...@oracle.com>>
> >>> À: "Stephen Colebourne" <scolebou...@joda.org 
> >>> <mailto:scolebou...@joda.org>>, "jigsaw-dev" <jigsaw-dev@openjdk.java.net 
> >>> <mailto:jigsaw-dev@openjdk.java.net>>
> >>> Envoyé: Mercredi 23 Mars 2016 16:18:50
> >>> Objet: Re: modulepath and classpath mixture
> >>>
> >>>
> >>> On 23/03/2016 14:42, Stephen Colebourne wrote:
> >>>> :
> >>>>
> >>>> I don't particularly care what the mechanism is for this, 

Re: modulepath and classpath mixture

[Alan Bateman]

On 30/03/2016 10:26, Stephen Colebourne wrote:

The underlying message of Jigsaw is that the classpath is going away.
There are no plans, at least not in this project, to remove the class 
path. There is of course a better world beyond the class path, it's just 
going to take a long time and require a lot of existing code and 
configuration to be cleaned up in order to get there. We have a good 
story for modules and the class path to co-exist and we expect they will 
co-exist for a long long time, maybe forever.


-Alan.

Re: modulepath and classpath mixture

[Ali Ebrahimi]

Also, we should look for solutions not workaround( or hacks) in remaining
time. I think classpath was that problem we trying avoid (or solve) with
jigsaw.

On Wed, Mar 30, 2016 at 1:56 PM, Stephen Colebourne <scolebou...@joda.org>
wrote:

> On 29 March 2016 at 17:33, Russell Gold <russell.g...@oracle.com> wrote:
> > Hi Stephen,
> > Why do you need any kind of friend access?
> > It seems to me that this is making things harder than they need to be.
> The tests can simply run with the production code on the class path, and
> then there are no module issues at all.
>
> The underlying message of Jigsaw is that the classpath is going away.
> An approach that requires use of the classpath is therefore not really
> a solution to the real problem.
>
> Stephen
>
>
> > I think a larger problem is that you can do what I just said with the
> jars, even a jar which has been designated as a module by virtue of having
> a module-info.class in it. That means that, when users are up taking jars,
> they are not prevented from accessing module internals. They can put the
> jars on the module path, of course, but they can still use them on the
> class path!
> >
> > - Russ
> >
> >>
> >> On 28 March 2016 at 11:13, Remi Forax <fo...@univ-mlv.fr> wrote:
> >>> Hi Stephen, Hi all,
> >>> I think that delivering tests as a separated module is a bad idea.
> >>>
> >>> I see that from the point of a developer, seeing the code and the test
> as different modules can be attractive because everything seems to be put
> at the right place but there is a big drawback. Because modules are reified
> at runtime, it means that the runtime environment of the tests will be
> different from the production environment.
> >>
> >> This last sentence doesn't make sense to me - tests are not run in a
> >> production environment.
> >>
> >> Tests have all the qualities of modules - code, dependencies,
> >> compilation phase, deployment. The only special part is the need for
> >> special "friend-like" access, which Jigsaw already has for other cases
> >> (the export...to clause).
> >>
> >> Put simply, I consider that module =
> >> deployment-artifact-with-dependencies. With that mental model, putting
> >> tests inside the module is just not acceptable, because tests should
> >> not be deployed with the main application and they have different
> >> dependencies. If we disagree that module =
> >> deployment-artifact-with-dependencies, then perhaps we have bigger
> >> problems to solve here.
> >>
> >> Stephen
> >> (And to Paul Benedict, the classpath is going to die over time, so any
> >> solution that uses that is flawed IMO).
> >>
> >>
> >>> So as Alan said, from the jigsaw point of view at runtime, the tests
> and the code should be in the same module.
> >>>
> >>> So the building tools have to come with a way to support to have 2
> different module-info.java in two different folders and package them as one
> module,
> >>> maybe javac should help by providing a way to merge 2 module-info at
> compile time.
> >>>
> >>> Rémi
> >>>
> >>> - Mail original -
> >>>> De: "Alan Bateman" <alan.bate...@oracle.com>
> >>>> À: "Stephen Colebourne" <scolebou...@joda.org>, "jigsaw-dev" <
> jigsaw-dev@openjdk.java.net>
> >>>> Envoyé: Mercredi 23 Mars 2016 16:18:50
> >>>> Objet: Re: modulepath and classpath mixture
> >>>>
> >>>>
> >>>> On 23/03/2016 14:42, Stephen Colebourne wrote:
> >>>>> :
> >>>>>
> >>>>> I don't particularly care what the mechanism is for this, but at the
> >>>>> requirements level:
> >>>>> - there are two modules - main and test
> >>>>> - each has its own source tree
> >>>>> - each has its own dependencies
> >>>>> - each is released separately
> >>>>> - each could be hosted on a central repo
> >>>>> - the test module needs to be able to contain the same packages as
> the
> >>>>> main module
> >>>>> - the test module needs to be able to invoke package-scoped code in
> >>>>> the same package in the main module
> >>>>>
> >>>>> To clarify further consider 4 modules, A, B, A-test and B-test where
> B
> >>>>&g

Re: modulepath and classpath mixture

[Stephen Colebourne]

On 29 March 2016 at 17:33, Russell Gold <russell.g...@oracle.com> wrote:
> Hi Stephen,
> Why do you need any kind of friend access?
> It seems to me that this is making things harder than they need to be. The 
> tests can simply run with the production code on the class path, and then 
> there are no module issues at all.

The underlying message of Jigsaw is that the classpath is going away.
An approach that requires use of the classpath is therefore not really
a solution to the real problem.

Stephen


> I think a larger problem is that you can do what I just said with the jars, 
> even a jar which has been designated as a module by virtue of having a 
> module-info.class in it. That means that, when users are up taking jars, they 
> are not prevented from accessing module internals. They can put the jars on 
> the module path, of course, but they can still use them on the class path!
>
> - Russ
>
>>
>> On 28 March 2016 at 11:13, Remi Forax <fo...@univ-mlv.fr> wrote:
>>> Hi Stephen, Hi all,
>>> I think that delivering tests as a separated module is a bad idea.
>>>
>>> I see that from the point of a developer, seeing the code and the test as 
>>> different modules can be attractive because everything seems to be put at 
>>> the right place but there is a big drawback. Because modules are reified at 
>>> runtime, it means that the runtime environment of the tests will be 
>>> different from the production environment.
>>
>> This last sentence doesn't make sense to me - tests are not run in a
>> production environment.
>>
>> Tests have all the qualities of modules - code, dependencies,
>> compilation phase, deployment. The only special part is the need for
>> special "friend-like" access, which Jigsaw already has for other cases
>> (the export...to clause).
>>
>> Put simply, I consider that module =
>> deployment-artifact-with-dependencies. With that mental model, putting
>> tests inside the module is just not acceptable, because tests should
>> not be deployed with the main application and they have different
>> dependencies. If we disagree that module =
>> deployment-artifact-with-dependencies, then perhaps we have bigger
>> problems to solve here.
>>
>> Stephen
>> (And to Paul Benedict, the classpath is going to die over time, so any
>> solution that uses that is flawed IMO).
>>
>>
>>> So as Alan said, from the jigsaw point of view at runtime, the tests and 
>>> the code should be in the same module.
>>>
>>> So the building tools have to come with a way to support to have 2 
>>> different module-info.java in two different folders and package them as one 
>>> module,
>>> maybe javac should help by providing a way to merge 2 module-info at 
>>> compile time.
>>>
>>> Rémi
>>>
>>> - Mail original -
>>>> De: "Alan Bateman" <alan.bate...@oracle.com>
>>>> À: "Stephen Colebourne" <scolebou...@joda.org>, "jigsaw-dev" 
>>>> <jigsaw-dev@openjdk.java.net>
>>>> Envoyé: Mercredi 23 Mars 2016 16:18:50
>>>> Objet: Re: modulepath and classpath mixture
>>>>
>>>>
>>>> On 23/03/2016 14:42, Stephen Colebourne wrote:
>>>>> :
>>>>>
>>>>> I don't particularly care what the mechanism is for this, but at the
>>>>> requirements level:
>>>>> - there are two modules - main and test
>>>>> - each has its own source tree
>>>>> - each has its own dependencies
>>>>> - each is released separately
>>>>> - each could be hosted on a central repo
>>>>> - the test module needs to be able to contain the same packages as the
>>>>> main module
>>>>> - the test module needs to be able to invoke package-scoped code in
>>>>> the same package in the main module
>>>>>
>>>>> To clarify further consider 4 modules, A, B, A-test and B-test where B
>>>>> depends on A. Module A-test may have a method foo() that uses package
>>>>> scope to access something in A. Module B-test will depend on A-test
>>>>> and rely on foo() to get access to that internal object.
>>>> To your list, I would add the ability to make use of testing frameworks
>>>> like TestNG and JUnit.
>>>>
>>>> In any case, and as things currently stand, you've got most of the
>>>> above. One differences is that the tests are not a separate module, they
>>>> are instead compiled and run in a way that patches the main module. The
>>>> second difference is that they don't have their own module declaration,
>>>> instead the compilation or run augments the dependences with any
>>>> additional dependences that the tests have. As I said, if they tools
>>>> makes it easy then I don't think it's too bad.
>>>>
>>>>>
>>>>> (Note that I view the thread discussion of
>>>>> references to test classes on the classpath as another hack.
>>>>>
>>>> Packages can't be split between modules and classpath so there is no
>>>> support for that.
>>>>
>>>> -Alan
>>>>
>

Re: modulepath and classpath mixture

[Paul Benedict]

Russell, when you drop a jar on the classpath, module code will not be able
to access it in a split package situation. That's the big barrier here.
Maven test projects are typically written with the same package shared with
the "main" code.

Cheers,
Paul

On Tue, Mar 29, 2016 at 11:33 AM, Russell Gold <russell.g...@oracle.com>
wrote:

> Hi Stephen,
>
> Why do you need any kind of friend access?
>
> It seems to me that this is making things harder than they need to be. The
> tests can simply run with the production code on the class path, and then
> there are no module issues at all.
>
> I think a larger problem is that you can do what I just said with the
> jars, even a jar which has been designated as a module by virtue of having
> a module-info.class in it. That means that, when users are up taking jars,
> they are not prevented from accessing module internals. They can put the
> jars on the module path, of course, but they can still use them on the
> class path!
>
> - Russ
>
> >
> > On 28 March 2016 at 11:13, Remi Forax <fo...@univ-mlv.fr> wrote:
> >> Hi Stephen, Hi all,
> >> I think that delivering tests as a separated module is a bad idea.
> >>
> >> I see that from the point of a developer, seeing the code and the test
> as different modules can be attractive because everything seems to be put
> at the right place but there is a big drawback. Because modules are reified
> at runtime, it means that the runtime environment of the tests will be
> different from the production environment.
> >
> > This last sentence doesn't make sense to me - tests are not run in a
> > production environment.
> >
> > Tests have all the qualities of modules - code, dependencies,
> > compilation phase, deployment. The only special part is the need for
> > special "friend-like" access, which Jigsaw already has for other cases
> > (the export...to clause).
> >
> > Put simply, I consider that module =
> > deployment-artifact-with-dependencies. With that mental model, putting
> > tests inside the module is just not acceptable, because tests should
> > not be deployed with the main application and they have different
> > dependencies. If we disagree that module =
> > deployment-artifact-with-dependencies, then perhaps we have bigger
> > problems to solve here.
> >
> > Stephen
> > (And to Paul Benedict, the classpath is going to die over time, so any
> > solution that uses that is flawed IMO).
> >
> >
> >> So as Alan said, from the jigsaw point of view at runtime, the tests
> and the code should be in the same module.
> >>
> >> So the building tools have to come with a way to support to have 2
> different module-info.java in two different folders and package them as one
> module,
> >> maybe javac should help by providing a way to merge 2 module-info at
> compile time.
> >>
> >> Rémi
> >>
> >> - Mail original -
> >>> De: "Alan Bateman" <alan.bate...@oracle.com>
> >>> À: "Stephen Colebourne" <scolebou...@joda.org>, "jigsaw-dev" <
> jigsaw-dev@openjdk.java.net>
> >>> Envoyé: Mercredi 23 Mars 2016 16:18:50
> >>> Objet: Re: modulepath and classpath mixture
> >>>
> >>>
> >>> On 23/03/2016 14:42, Stephen Colebourne wrote:
> >>>> :
> >>>>
> >>>> I don't particularly care what the mechanism is for this, but at the
> >>>> requirements level:
> >>>> - there are two modules - main and test
> >>>> - each has its own source tree
> >>>> - each has its own dependencies
> >>>> - each is released separately
> >>>> - each could be hosted on a central repo
> >>>> - the test module needs to be able to contain the same packages as the
> >>>> main module
> >>>> - the test module needs to be able to invoke package-scoped code in
> >>>> the same package in the main module
> >>>>
> >>>> To clarify further consider 4 modules, A, B, A-test and B-test where B
> >>>> depends on A. Module A-test may have a method foo() that uses package
> >>>> scope to access something in A. Module B-test will depend on A-test
> >>>> and rely on foo() to get access to that internal object.
> >>> To your list, I would add the ability to make use of testing frameworks
> >>> like TestNG and JUnit.
> >>>
> >>> In any case, and as things currently stand, you've got most of the
> >>> above. One differences is that the tests are not a separate module,
> they
> >>> are instead compiled and run in a way that patches the main module. The
> >>> second difference is that they don't have their own module declaration,
> >>> instead the compilation or run augments the dependences with any
> >>> additional dependences that the tests have. As I said, if they tools
> >>> makes it easy then I don't think it's too bad.
> >>>
> >>>>
> >>>> (Note that I view the thread discussion of
> >>>> references to test classes on the classpath as another hack.
> >>>>
> >>> Packages can't be split between modules and classpath so there is no
> >>> support for that.
> >>>
> >>> -Alan
> >>>
>
>

Re: modulepath and classpath mixture

[Russell Gold]

Hi Stephen,

Why do you need any kind of friend access?

It seems to me that this is making things harder than they need to be. The 
tests can simply run with the production code on the class path, and then there 
are no module issues at all. 

I think a larger problem is that you can do what I just said with the jars, 
even a jar which has been designated as a module by virtue of having a 
module-info.class in it. That means that, when users are up taking jars, they 
are not prevented from accessing module internals. They can put the jars on the 
module path, of course, but they can still use them on the class path!

- Russ

> 
> On 28 March 2016 at 11:13, Remi Forax <fo...@univ-mlv.fr> wrote:
>> Hi Stephen, Hi all,
>> I think that delivering tests as a separated module is a bad idea.
>> 
>> I see that from the point of a developer, seeing the code and the test as 
>> different modules can be attractive because everything seems to be put at 
>> the right place but there is a big drawback. Because modules are reified at 
>> runtime, it means that the runtime environment of the tests will be 
>> different from the production environment.
> 
> This last sentence doesn't make sense to me - tests are not run in a
> production environment.
> 
> Tests have all the qualities of modules - code, dependencies,
> compilation phase, deployment. The only special part is the need for
> special "friend-like" access, which Jigsaw already has for other cases
> (the export...to clause).
> 
> Put simply, I consider that module =
> deployment-artifact-with-dependencies. With that mental model, putting
> tests inside the module is just not acceptable, because tests should
> not be deployed with the main application and they have different
> dependencies. If we disagree that module =
> deployment-artifact-with-dependencies, then perhaps we have bigger
> problems to solve here.
> 
> Stephen
> (And to Paul Benedict, the classpath is going to die over time, so any
> solution that uses that is flawed IMO).
> 
> 
>> So as Alan said, from the jigsaw point of view at runtime, the tests and the 
>> code should be in the same module.
>> 
>> So the building tools have to come with a way to support to have 2 different 
>> module-info.java in two different folders and package them as one module,
>> maybe javac should help by providing a way to merge 2 module-info at compile 
>> time.
>> 
>> Rémi
>> 
>> - Mail original -
>>> De: "Alan Bateman" <alan.bate...@oracle.com>
>>> À: "Stephen Colebourne" <scolebou...@joda.org>, "jigsaw-dev" 
>>> <jigsaw-dev@openjdk.java.net>
>>> Envoyé: Mercredi 23 Mars 2016 16:18:50
>>> Objet: Re: modulepath and classpath mixture
>>> 
>>> 
>>> On 23/03/2016 14:42, Stephen Colebourne wrote:
>>>> :
>>>> 
>>>> I don't particularly care what the mechanism is for this, but at the
>>>> requirements level:
>>>> - there are two modules - main and test
>>>> - each has its own source tree
>>>> - each has its own dependencies
>>>> - each is released separately
>>>> - each could be hosted on a central repo
>>>> - the test module needs to be able to contain the same packages as the
>>>> main module
>>>> - the test module needs to be able to invoke package-scoped code in
>>>> the same package in the main module
>>>> 
>>>> To clarify further consider 4 modules, A, B, A-test and B-test where B
>>>> depends on A. Module A-test may have a method foo() that uses package
>>>> scope to access something in A. Module B-test will depend on A-test
>>>> and rely on foo() to get access to that internal object.
>>> To your list, I would add the ability to make use of testing frameworks
>>> like TestNG and JUnit.
>>> 
>>> In any case, and as things currently stand, you've got most of the
>>> above. One differences is that the tests are not a separate module, they
>>> are instead compiled and run in a way that patches the main module. The
>>> second difference is that they don't have their own module declaration,
>>> instead the compilation or run augments the dependences with any
>>> additional dependences that the tests have. As I said, if they tools
>>> makes it easy then I don't think it's too bad.
>>> 
>>>> 
>>>> (Note that I view the thread discussion of
>>>> references to test classes on the classpath as another hack.
>>>> 
>>> Packages can't be split between modules and classpath so there is no
>>> support for that.
>>> 
>>> -Alan
>>> 

Re: modulepath and classpath mixture

[forax]

Paul, 
you can give a different version to your module but you don't have to. 
Test can use the very same module name along with the same version. 

Rémi 

- Mail original -

> De: "Paul Benedict" <pbened...@apache.org>
> À: "Remi Forax" <fo...@univ-mlv.fr>
> Cc: "Stephen Colebourne" <scolebou...@joda.org>, "jigsaw-dev"
> <jigsaw-dev@openjdk.java.net>
> Envoyé: Mardi 29 Mars 2016 17:55:10
> Objet: Re: modulepath and classpath mixture

> Remi, it's really untenable to give tests a different version. Tests are
> currently built under the same version as the main project under one master
> parent project. Split packages are still going to be a necessary because
> that's the world of thousands of existing projects. So that might be against
> jigsaw canon but hopefully a compromise can be found.

> Cheers,
> Paul

> On Tue, Mar 29, 2016 at 10:41 AM, Remi Forax < fo...@univ-mlv.fr > wrote:

> > - Mail original -
> 
> > > De: "Stephen Colebourne" < scolebou...@joda.org >
> 
> > > À: "jigsaw-dev" < jigsaw-dev@openjdk.java.net >
> 
> > > Envoyé: Mardi 29 Mars 2016 15:24:15
> 
> > > Objet: Re: modulepath and classpath mixture
> 
> > >
> 
> > > On 28 March 2016 at 11:13, Remi Forax < fo...@univ-mlv.fr > wrote:
> 
> > > > Hi Stephen, Hi all,
> 
> > > > I think that delivering tests as a separated module is a bad idea.
> 
> > > >
> 
> > > > I see that from the point of a developer, seeing the code and the test
> > > > as
> 
> > > > different modules can be attractive because everything seems to be put
> > > > at
> 
> > > > the right place but there is a big drawback. Because modules are
> > > > reified
> 
> > > > at runtime, it means that the runtime environment of the tests will be
> 
> > > > different from the production environment.
> 
> > >
> 
> > > This last sentence doesn't make sense to me - tests are not run in a
> 
> > > production environment.
> 

> > yes, you just want your tests to run in a runtime which is as close as
> > possible to the runtime of the production environment.
> 
> > It's like when a test fails because JUnit calls the test concurrently when
> > in
> > production the code always run under a giant lock.
> 

> > >
> 
> > > Tests have all the qualities of modules - code, dependencies,
> 
> > > compilation phase, deployment. The only special part is the need for
> 
> > > special "friend-like" access, which Jigsaw already has for other cases
> 
> > > (the export...to clause).
> 

> > You also need to support split packages or a kind of export...to that
> > allows
> > to export package private method as public.
> 
> > These are two major deviations from the jigsaw canon.
> 

> > Instead of considering tests as a separated module, i prefer to consider
> > them
> > as a separated version (1.0 vs 1.0-test),
> 
> > the test version includes the code, the tests, and have several more
> > dependencies on JUnit like modules.
> 

> > >
> 
> > > Put simply, I consider that module =
> 
> > > deployment-artifact-with-dependencies. With that mental model, putting
> 
> > > tests inside the module is just not acceptable, because tests should
> 
> > > not be deployed with the main application and they have different
> 
> > > dependencies. If we disagree that module =
> 
> > > deployment-artifact-with-dependencies, then perhaps we have bigger
> 
> > > problems to solve here.
> 

> > We agree that a module = deployment-artifact-with-dependencies, but we
> > disagree because for me the test should be a module with the same name but
> > a
> > different version (by example with a test suffix) while you think it should
> > be a different module.
> 

> > >
> 
> > > Stephen
> 
> > > (And to Paul Benedict, the classpath is going to die over time, so any
> 
> > > solution that uses that is flawed IMO).
> 

> > I don't think the classpath will die but if you want to use things like
> > jlink, it will not work if you have jars in the a classpath.
> 

> > Rémi
> 

> > >
> 
> > >
> 
> > > > So as Alan said, from the jigsaw point of view at runtime, the tests
> > > > and
> 
> > > > the code should be in the same module.
> 
> > > >
>

Re: modulepath and classpath mixture

[Paul Benedict]

Remi, it's really untenable to give tests a different version. Tests are
currently built under the same version as the main project under one master
parent project. Split packages are still going to be a necessary because
that's the world of thousands of existing projects. So that might be
against jigsaw canon but hopefully a compromise can be found.

Cheers,
Paul

On Tue, Mar 29, 2016 at 10:41 AM, Remi Forax <fo...@univ-mlv.fr> wrote:

>
>
> - Mail original -
> > De: "Stephen Colebourne" <scolebou...@joda.org>
> > À: "jigsaw-dev" <jigsaw-dev@openjdk.java.net>
> > Envoyé: Mardi 29 Mars 2016 15:24:15
> > Objet: Re: modulepath and classpath mixture
> >
> > On 28 March 2016 at 11:13, Remi Forax <fo...@univ-mlv.fr> wrote:
> > > Hi Stephen, Hi all,
> > > I think that delivering tests as a separated module is a bad idea.
> > >
> > > I see that from the point of a developer, seeing the code and the test
> as
> > > different modules can be attractive because everything seems to be put
> at
> > > the right place but there is a big drawback. Because modules are
> reified
> > > at runtime, it means that the runtime environment of the tests will be
> > > different from the production environment.
> >
> > This last sentence doesn't make sense to me - tests are not run in a
> > production environment.
>
> yes, you just want your tests to run in a runtime which is as close as
> possible to the runtime of the production environment.
> It's like when a test fails because JUnit calls the test concurrently when
> in production the code always run under a giant lock.
>
> >
> > Tests have all the qualities of modules - code, dependencies,
> > compilation phase, deployment. The only special part is the need for
> > special "friend-like" access, which Jigsaw already has for other cases
> > (the export...to clause).
>
> You also need to support split packages or a kind of export...to that
> allows to export package private method as public.
> These are two major deviations from the jigsaw canon.
>
> Instead of considering tests as a separated module, i prefer to consider
> them as a separated version (1.0 vs 1.0-test),
> the test version includes the code, the tests, and have several more
> dependencies on JUnit like modules.
>
> >
> > Put simply, I consider that module =
> > deployment-artifact-with-dependencies. With that mental model, putting
> > tests inside the module is just not acceptable, because tests should
> > not be deployed with the main application and they have different
> > dependencies. If we disagree that module =
> > deployment-artifact-with-dependencies, then perhaps we have bigger
> > problems to solve here.
>
> We agree that a module = deployment-artifact-with-dependencies, but we
> disagree because for me the test should be a module with the same name but
> a different version (by example with a test suffix) while you think it
> should be a different module.
>
> >
> > Stephen
> > (And to Paul Benedict, the classpath is going to die over time, so any
> > solution that uses that is flawed IMO).
>
> I don't think the classpath will die but if you want to use things like
> jlink, it will not work if you have jars in the a classpath.
>
> Rémi
>
> >
> >
> > > So as Alan said, from the jigsaw point of view at runtime, the tests
> and
> > > the code should be in the same module.
> > >
> > > So the building tools have to come with a way to support to have 2
> > > different module-info.java in two different folders and package them as
> > > one module,
> > > maybe javac should help by providing a way to merge 2 module-info at
> > > compile time.
> > >
> > > Rémi
> > >
> > > - Mail original -
> > >> De: "Alan Bateman" <alan.bate...@oracle.com>
> > >> À: "Stephen Colebourne" <scolebou...@joda.org>, "jigsaw-dev"
> > >> <jigsaw-dev@openjdk.java.net>
> > >> Envoyé: Mercredi 23 Mars 2016 16:18:50
> > >> Objet: Re: modulepath and classpath mixture
> > >>
> > >>
> > >> On 23/03/2016 14:42, Stephen Colebourne wrote:
> > >> > :
> > >> >
> > >> > I don't particularly care what the mechanism is for this, but at the
> > >> > requirements level:
> > >> > - there are two modules - main and test
> > >> > - each has its own source tree
> > >> > - each has its

Re: modulepath and classpath mixture

[Remi Forax]

- Mail original -
> De: "Stephen Colebourne" <scolebou...@joda.org>
> À: "jigsaw-dev" <jigsaw-dev@openjdk.java.net>
> Envoyé: Mardi 29 Mars 2016 15:24:15
> Objet: Re: modulepath and classpath mixture
> 
> On 28 March 2016 at 11:13, Remi Forax <fo...@univ-mlv.fr> wrote:
> > Hi Stephen, Hi all,
> > I think that delivering tests as a separated module is a bad idea.
> >
> > I see that from the point of a developer, seeing the code and the test as
> > different modules can be attractive because everything seems to be put at
> > the right place but there is a big drawback. Because modules are reified
> > at runtime, it means that the runtime environment of the tests will be
> > different from the production environment.
> 
> This last sentence doesn't make sense to me - tests are not run in a
> production environment.

yes, you just want your tests to run in a runtime which is as close as possible 
to the runtime of the production environment.
It's like when a test fails because JUnit calls the test concurrently when in 
production the code always run under a giant lock. 

> 
> Tests have all the qualities of modules - code, dependencies,
> compilation phase, deployment. The only special part is the need for
> special "friend-like" access, which Jigsaw already has for other cases
> (the export...to clause).

You also need to support split packages or a kind of export...to that allows to 
export package private method as public.
These are two major deviations from the jigsaw canon. 

Instead of considering tests as a separated module, i prefer to consider them 
as a separated version (1.0 vs 1.0-test),
the test version includes the code, the tests, and have several more 
dependencies on JUnit like modules. 

> 
> Put simply, I consider that module =
> deployment-artifact-with-dependencies. With that mental model, putting
> tests inside the module is just not acceptable, because tests should
> not be deployed with the main application and they have different
> dependencies. If we disagree that module =
> deployment-artifact-with-dependencies, then perhaps we have bigger
> problems to solve here.

We agree that a module = deployment-artifact-with-dependencies, but we disagree 
because for me the test should be a module with the same name but a different 
version (by example with a test suffix) while you think it should be a 
different module.

> 
> Stephen
> (And to Paul Benedict, the classpath is going to die over time, so any
> solution that uses that is flawed IMO).

I don't think the classpath will die but if you want to use things like jlink, 
it will not work if you have jars in the a classpath.

Rémi

> 
> 
> > So as Alan said, from the jigsaw point of view at runtime, the tests and
> > the code should be in the same module.
> >
> > So the building tools have to come with a way to support to have 2
> > different module-info.java in two different folders and package them as
> > one module,
> > maybe javac should help by providing a way to merge 2 module-info at
> > compile time.
> >
> > Rémi
> >
> > - Mail original -
> >> De: "Alan Bateman" <alan.bate...@oracle.com>
> >> À: "Stephen Colebourne" <scolebou...@joda.org>, "jigsaw-dev"
> >> <jigsaw-dev@openjdk.java.net>
> >> Envoyé: Mercredi 23 Mars 2016 16:18:50
> >> Objet: Re: modulepath and classpath mixture
> >>
> >>
> >> On 23/03/2016 14:42, Stephen Colebourne wrote:
> >> > :
> >> >
> >> > I don't particularly care what the mechanism is for this, but at the
> >> > requirements level:
> >> > - there are two modules - main and test
> >> > - each has its own source tree
> >> > - each has its own dependencies
> >> > - each is released separately
> >> > - each could be hosted on a central repo
> >> > - the test module needs to be able to contain the same packages as the
> >> > main module
> >> > - the test module needs to be able to invoke package-scoped code in
> >> > the same package in the main module
> >> >
> >> > To clarify further consider 4 modules, A, B, A-test and B-test where B
> >> > depends on A. Module A-test may have a method foo() that uses package
> >> > scope to access something in A. Module B-test will depend on A-test
> >> > and rely on foo() to get access to that internal object.
> >> To your list, I would add the ability to make use of testing frameworks
> >> like TestNG and JUnit.
> >>
> >> In any case, and as t

Re: modulepath and classpath mixture

[Stephen Colebourne]

On 28 March 2016 at 11:13, Remi Forax <fo...@univ-mlv.fr> wrote:
> Hi Stephen, Hi all,
> I think that delivering tests as a separated module is a bad idea.
>
> I see that from the point of a developer, seeing the code and the test as 
> different modules can be attractive because everything seems to be put at the 
> right place but there is a big drawback. Because modules are reified at 
> runtime, it means that the runtime environment of the tests will be different 
> from the production environment.

This last sentence doesn't make sense to me - tests are not run in a
production environment.

Tests have all the qualities of modules - code, dependencies,
compilation phase, deployment. The only special part is the need for
special "friend-like" access, which Jigsaw already has for other cases
(the export...to clause).

Put simply, I consider that module =
deployment-artifact-with-dependencies. With that mental model, putting
tests inside the module is just not acceptable, because tests should
not be deployed with the main application and they have different
dependencies. If we disagree that module =
deployment-artifact-with-dependencies, then perhaps we have bigger
problems to solve here.

Stephen
(And to Paul Benedict, the classpath is going to die over time, so any
solution that uses that is flawed IMO).


> So as Alan said, from the jigsaw point of view at runtime, the tests and the 
> code should be in the same module.
>
> So the building tools have to come with a way to support to have 2 different 
> module-info.java in two different folders and package them as one module,
> maybe javac should help by providing a way to merge 2 module-info at compile 
> time.
>
> Rémi
>
> - Mail original -
>> De: "Alan Bateman" <alan.bate...@oracle.com>
>> À: "Stephen Colebourne" <scolebou...@joda.org>, "jigsaw-dev" 
>> <jigsaw-dev@openjdk.java.net>
>> Envoyé: Mercredi 23 Mars 2016 16:18:50
>> Objet: Re: modulepath and classpath mixture
>>
>>
>> On 23/03/2016 14:42, Stephen Colebourne wrote:
>> > :
>> >
>> > I don't particularly care what the mechanism is for this, but at the
>> > requirements level:
>> > - there are two modules - main and test
>> > - each has its own source tree
>> > - each has its own dependencies
>> > - each is released separately
>> > - each could be hosted on a central repo
>> > - the test module needs to be able to contain the same packages as the
>> > main module
>> > - the test module needs to be able to invoke package-scoped code in
>> > the same package in the main module
>> >
>> > To clarify further consider 4 modules, A, B, A-test and B-test where B
>> > depends on A. Module A-test may have a method foo() that uses package
>> > scope to access something in A. Module B-test will depend on A-test
>> > and rely on foo() to get access to that internal object.
>> To your list, I would add the ability to make use of testing frameworks
>> like TestNG and JUnit.
>>
>> In any case, and as things currently stand, you've got most of the
>> above. One differences is that the tests are not a separate module, they
>> are instead compiled and run in a way that patches the main module. The
>> second difference is that they don't have their own module declaration,
>> instead the compilation or run augments the dependences with any
>> additional dependences that the tests have. As I said, if they tools
>> makes it easy then I don't think it's too bad.
>>
>> >
>> > (Note that I view the thread discussion of
>> > references to test classes on the classpath as another hack.
>> >
>> Packages can't be split between modules and classpath so there is no
>> support for that.
>>
>> -Alan
>>

Re: modulepath and classpath mixture

[Paul Benedict]

Stephen C, I'd like to question the assumption why tests must be their own
module? Clearly tests are going to be their own artifact, but I am not sure
they need the semantics of module boundaries. Placing on them classpath
would be the best thing for them, I think. However, two requirements would
still need to be met:

(a) Split packages need to be allowed so test classes can live in the same
package as the main classes
(b) Test classes can replace main classes

The patch/override command line argument already satisfies (b). Regarding
(a), although the classpath exports everything, code in an named module
can't access types in the unnamed module. That will be necessary so
existing code in "main" can access the replacement classes in "test".

If there was a mechanism to allow classpath code to overlay with module
code, existing Maven projects could function normally.

Cheers,
Paul

On Mon, Mar 28, 2016 at 9:23 AM, Stephen Felts <stephen.fe...@oracle.com>
wrote:

> Having spent 3 years using OSGI, fragment bundles were a great feature
> that we used to overcome this type of problem and others.
> I like the idea but I don't think it will be popular in Jigsaw (and OSGI
> is a four-letter word).
>
>
>
> -Original Message-
> From: Ali Ebrahimi [mailto:ali.ebrahimi1...@gmail.com]
> Sent: Monday, March 28, 2016 8:56 AM
> To: Remi Forax
> Cc: jigsaw-dev
> Subject: Re: modulepath and classpath mixture
>
> I think we can adapt OSGI fragment bundle here and introduce module
> fragments that is attached to a host module.
>
> So we would have test fragment that is embeded in main module. So we would
> have one module.
> What do you think?
>
> On Mon, Mar 28, 2016 at 2:43 PM, Remi Forax <fo...@univ-mlv.fr> wrote:
>
> > Hi Stephen, Hi all,
> > I think that delivering tests as a separated module is a bad idea.
> >
> > I see that from the point of a developer, seeing the code and the test
> > as different modules can be attractive because everything seems to be
> > put at the right place but there is a big drawback. Because modules
> > are reified at runtime, it means that the runtime environment of the
> > tests will be different from the production environment.
> >
> > So as Alan said, from the jigsaw point of view at runtime, the tests
> > and the code should be in the same module.
> >
> > So the building tools have to come with a way to support to have 2
> > different module-info.java in two different folders and package them
> > as one module, maybe javac should help by providing a way to merge 2
> > module-info at compile time.
> >
> > Rémi
> >
> > - Mail original -
> > > De: "Alan Bateman" <alan.bate...@oracle.com>
> > > À: "Stephen Colebourne" <scolebou...@joda.org>, "jigsaw-dev" <
> > jigsaw-dev@openjdk.java.net>
> > > Envoyé: Mercredi 23 Mars 2016 16:18:50
> > > Objet: Re: modulepath and classpath mixture
> > >
> > >
> > > On 23/03/2016 14:42, Stephen Colebourne wrote:
> > > > :
> > > >
> > > > I don't particularly care what the mechanism is for this, but at
> > > > the requirements level:
> > > > - there are two modules - main and test
> > > > - each has its own source tree
> > > > - each has its own dependencies
> > > > - each is released separately
> > > > - each could be hosted on a central repo
> > > > - the test module needs to be able to contain the same packages as
> > > > the main module
> > > > - the test module needs to be able to invoke package-scoped code
> > > > in the same package in the main module
> > > >
> > > > To clarify further consider 4 modules, A, B, A-test and B-test
> > > > where B depends on A. Module A-test may have a method foo() that
> > > > uses package scope to access something in A. Module B-test will
> > > > depend on A-test and rely on foo() to get access to that internal
> object.
> > > To your list, I would add the ability to make use of testing
> > > frameworks like TestNG and JUnit.
> > >
> > > In any case, and as things currently stand, you've got most of the
> > > above. One differences is that the tests are not a separate module,
> > > they are instead compiled and run in a way that patches the main
> > > module. The second difference is that they don't have their own
> > > module declaration, instead the compilation or run augments the
> > > dependences with any additional dependences that the tests have. As
> > > I said, if they tools makes it easy then I don't think it's too bad.
> > >
> > > >
> > > > (Note that I view the thread discussion of references to test
> > > > classes on the classpath as another hack.
> > > >
> > > Packages can't be split between modules and classpath so there is no
> > > support for that.
> > >
> > > -Alan
> > >
> >
>
>
>
> --
>
> Best Regards,
> Ali Ebrahimi
>

RE: modulepath and classpath mixture

[Stephen Felts]

Having spent 3 years using OSGI, fragment bundles were a great feature that we 
used to overcome this type of problem and others.
I like the idea but I don't think it will be popular in Jigsaw (and OSGI is a 
four-letter word).



-Original Message-
From: Ali Ebrahimi [mailto:ali.ebrahimi1...@gmail.com] 
Sent: Monday, March 28, 2016 8:56 AM
To: Remi Forax
Cc: jigsaw-dev
Subject: Re: modulepath and classpath mixture

I think we can adapt OSGI fragment bundle here and introduce module fragments 
that is attached to a host module.

So we would have test fragment that is embeded in main module. So we would have 
one module.
What do you think?

On Mon, Mar 28, 2016 at 2:43 PM, Remi Forax <fo...@univ-mlv.fr> wrote:

> Hi Stephen, Hi all,
> I think that delivering tests as a separated module is a bad idea.
>
> I see that from the point of a developer, seeing the code and the test 
> as different modules can be attractive because everything seems to be 
> put at the right place but there is a big drawback. Because modules 
> are reified at runtime, it means that the runtime environment of the 
> tests will be different from the production environment.
>
> So as Alan said, from the jigsaw point of view at runtime, the tests 
> and the code should be in the same module.
>
> So the building tools have to come with a way to support to have 2 
> different module-info.java in two different folders and package them 
> as one module, maybe javac should help by providing a way to merge 2 
> module-info at compile time.
>
> Rémi
>
> - Mail original -
> > De: "Alan Bateman" <alan.bate...@oracle.com>
> > À: "Stephen Colebourne" <scolebou...@joda.org>, "jigsaw-dev" <
> jigsaw-dev@openjdk.java.net>
> > Envoyé: Mercredi 23 Mars 2016 16:18:50
> > Objet: Re: modulepath and classpath mixture
> >
> >
> > On 23/03/2016 14:42, Stephen Colebourne wrote:
> > > :
> > >
> > > I don't particularly care what the mechanism is for this, but at 
> > > the requirements level:
> > > - there are two modules - main and test
> > > - each has its own source tree
> > > - each has its own dependencies
> > > - each is released separately
> > > - each could be hosted on a central repo
> > > - the test module needs to be able to contain the same packages as 
> > > the main module
> > > - the test module needs to be able to invoke package-scoped code 
> > > in the same package in the main module
> > >
> > > To clarify further consider 4 modules, A, B, A-test and B-test 
> > > where B depends on A. Module A-test may have a method foo() that 
> > > uses package scope to access something in A. Module B-test will 
> > > depend on A-test and rely on foo() to get access to that internal object.
> > To your list, I would add the ability to make use of testing 
> > frameworks like TestNG and JUnit.
> >
> > In any case, and as things currently stand, you've got most of the 
> > above. One differences is that the tests are not a separate module, 
> > they are instead compiled and run in a way that patches the main 
> > module. The second difference is that they don't have their own 
> > module declaration, instead the compilation or run augments the 
> > dependences with any additional dependences that the tests have. As 
> > I said, if they tools makes it easy then I don't think it's too bad.
> >
> > >
> > > (Note that I view the thread discussion of references to test 
> > > classes on the classpath as another hack.
> > >
> > Packages can't be split between modules and classpath so there is no 
> > support for that.
> >
> > -Alan
> >
>



-- 

Best Regards,
Ali Ebrahimi

Re: modulepath and classpath mixture

[Ali Ebrahimi]

I think we can adapt OSGI fragment bundle here and introduce module
fragments that is attached to a host module.

So we would have test fragment that is embeded in main module. So we would
have one module.
What do you think?

On Mon, Mar 28, 2016 at 2:43 PM, Remi Forax <fo...@univ-mlv.fr> wrote:

> Hi Stephen, Hi all,
> I think that delivering tests as a separated module is a bad idea.
>
> I see that from the point of a developer, seeing the code and the test as
> different modules can be attractive because everything seems to be put at
> the right place but there is a big drawback. Because modules are reified at
> runtime, it means that the runtime environment of the tests will be
> different from the production environment.
>
> So as Alan said, from the jigsaw point of view at runtime, the tests and
> the code should be in the same module.
>
> So the building tools have to come with a way to support to have 2
> different module-info.java in two different folders and package them as one
> module,
> maybe javac should help by providing a way to merge 2 module-info at
> compile time.
>
> Rémi
>
> - Mail original -
> > De: "Alan Bateman" <alan.bate...@oracle.com>
> > À: "Stephen Colebourne" <scolebou...@joda.org>, "jigsaw-dev" <
> jigsaw-dev@openjdk.java.net>
> > Envoyé: Mercredi 23 Mars 2016 16:18:50
> > Objet: Re: modulepath and classpath mixture
> >
> >
> > On 23/03/2016 14:42, Stephen Colebourne wrote:
> > > :
> > >
> > > I don't particularly care what the mechanism is for this, but at the
> > > requirements level:
> > > - there are two modules - main and test
> > > - each has its own source tree
> > > - each has its own dependencies
> > > - each is released separately
> > > - each could be hosted on a central repo
> > > - the test module needs to be able to contain the same packages as the
> > > main module
> > > - the test module needs to be able to invoke package-scoped code in
> > > the same package in the main module
> > >
> > > To clarify further consider 4 modules, A, B, A-test and B-test where B
> > > depends on A. Module A-test may have a method foo() that uses package
> > > scope to access something in A. Module B-test will depend on A-test
> > > and rely on foo() to get access to that internal object.
> > To your list, I would add the ability to make use of testing frameworks
> > like TestNG and JUnit.
> >
> > In any case, and as things currently stand, you've got most of the
> > above. One differences is that the tests are not a separate module, they
> > are instead compiled and run in a way that patches the main module. The
> > second difference is that they don't have their own module declaration,
> > instead the compilation or run augments the dependences with any
> > additional dependences that the tests have. As I said, if they tools
> > makes it easy then I don't think it's too bad.
> >
> > >
> > > (Note that I view the thread discussion of
> > > references to test classes on the classpath as another hack.
> > >
> > Packages can't be split between modules and classpath so there is no
> > support for that.
> >
> > -Alan
> >
>



-- 

Best Regards,
Ali Ebrahimi

Re: modulepath and classpath mixture

[Remi Forax]

Hi Stephen, Hi all,
I think that delivering tests as a separated module is a bad idea.

I see that from the point of a developer, seeing the code and the test as 
different modules can be attractive because everything seems to be put at the 
right place but there is a big drawback. Because modules are reified at 
runtime, it means that the runtime environment of the tests will be different 
from the production environment.

So as Alan said, from the jigsaw point of view at runtime, the tests and the 
code should be in the same module.

So the building tools have to come with a way to support to have 2 different 
module-info.java in two different folders and package them as one module,
maybe javac should help by providing a way to merge 2 module-info at compile 
time.

Rémi

- Mail original -
> De: "Alan Bateman" <alan.bate...@oracle.com>
> À: "Stephen Colebourne" <scolebou...@joda.org>, "jigsaw-dev" 
> <jigsaw-dev@openjdk.java.net>
> Envoyé: Mercredi 23 Mars 2016 16:18:50
> Objet: Re: modulepath and classpath mixture
> 
> 
> On 23/03/2016 14:42, Stephen Colebourne wrote:
> > :
> >
> > I don't particularly care what the mechanism is for this, but at the
> > requirements level:
> > - there are two modules - main and test
> > - each has its own source tree
> > - each has its own dependencies
> > - each is released separately
> > - each could be hosted on a central repo
> > - the test module needs to be able to contain the same packages as the
> > main module
> > - the test module needs to be able to invoke package-scoped code in
> > the same package in the main module
> >
> > To clarify further consider 4 modules, A, B, A-test and B-test where B
> > depends on A. Module A-test may have a method foo() that uses package
> > scope to access something in A. Module B-test will depend on A-test
> > and rely on foo() to get access to that internal object.
> To your list, I would add the ability to make use of testing frameworks
> like TestNG and JUnit.
> 
> In any case, and as things currently stand, you've got most of the
> above. One differences is that the tests are not a separate module, they
> are instead compiled and run in a way that patches the main module. The
> second difference is that they don't have their own module declaration,
> instead the compilation or run augments the dependences with any
> additional dependences that the tests have. As I said, if they tools
> makes it easy then I don't think it's too bad.
> 
> >
> > (Note that I view the thread discussion of
> > references to test classes on the classpath as another hack.
> >
> Packages can't be split between modules and classpath so there is no
> support for that.
> 
> -Alan
> 

Re: modulepath and classpath mixture

[Russell Gold]

> On Mar 23, 2016, at 11:42 AM, Alan Bateman  wrote:
> 
> 
> 
> On 23/03/2016 14:15, Russell Gold wrote:
>> Here are my assumptions, which you can correct.
>> 
>> 1. A jar or classes directory placed on a class path are treated as part of 
>> the unnamed module
> Yes

Then it seems to me that there isn’t actually a problem, here. Build tools can 
continue to work unchanged, using class paths to allow unit tests to access the 
code they are testing. The jars are then treated as modules by other builds, 
which shouldn’t be accessing the module internals in any case. If they really 
do need to access them, they just put those jars on their class paths - but 
that’s a hint of poor modularity and testability.

> 
>> 2. A jar containing a module-info file is treated as a module when placed on 
>> a module path, and restricts access to non-exported packages
> It also declares what it depends on and maybe services that it uses/provides.
> 
> Also a JAR file on the module path without a module-info.class is treated as 
> an automatic module.

And what is an automatic module? I didn’t see that in the specs
> 
> 
>> 3. A jmod file is only usable on the module path.
> JMOD files are unlikely to be seen by most developers. They are not an 
> executable format. They are mostly for link-time when creating a custom 
> run-time image.
> 
> 
>> 
>> If a class directory or a jar function as modules when placed on the class 
>> path, what is the point of a separate module path? And even in that case, 
>> doesn’t it make sense to treat a class directory as part of the unnamed 
>> module in general? That makes this problem go away completely. It doesn’t 
>> provide access to third parties, as code would not be distributed as loose 
>> classes.
>> 
> You would loose the benefits of the module path and I assume couldn't 
> distinguish automatic modules from regular JAR files. In general I would 
> think this would be just confusing and I don't know what problem it would 
> solve.

It doesn’t, because it was based on my assumptions being wrong :)

I was trying to understand the problems Robert and Stephen were raising - and I 
cannot see the issue. 

- Russ

Re: modulepath and classpath mixture

[Stephen Colebourne]

On 23 March 2016 at 12:51, Alan Bateman  wrote:
> If types T1 and T2 have the same defining loader and both types are in the
> same package then they are in the same module. T1 can't be module M1 and T2
> in a different module M2.
> (I shudder the thought of it being different or attempting to change what
> default/package access means after all these years.)

I'm not asking for package scope to change meaning. It is Jigsaw that
is trying to change its meaning wrt tests.

Discussion of white vs black box testing is all very well and good,
but its completely impractical. There are tens of thousands of
existing projects out there, in open source and private, that rely on
tests being in the same package. It is in essence a fundamental part
of common approaches to testing. Making it hard to test code in the
same package, as seems to be proposed, would be a nightmare for
migration. (Many open source projects are for fun in spare time. Who
is going to want to refactor all their tests to meet what amounts to
an artificial limitation? Not me.)

I don't particularly care what the mechanism is for this, but at the
requirements level:
- there are two modules - main and test
- each has its own source tree
- each has its own dependencies
- each is released separately
- each could be hosted on a central repo
- the test module needs to be able to contain the same packages as the
main module
- the test module needs to be able to invoke package-scoped code in
the same package in the main module

To clarify further consider 4 modules, A, B, A-test and B-test where B
depends on A. Module A-test may have a method foo() that uses package
scope to access something in A. Module B-test will depend on A-test
and rely on foo() to get access to that internal object.

One apparent option would appear to be to merge the two modules
dynamically at load time. However, they do need to be released and
published separately (Note that I view the thread discussion of
references to test classes on the classpath as another hack. The
release and dependency elements clearly show tests to be modules, just
ones with special access.)

Stephen

Re: modulepath and classpath mixture

[Alan Bateman]

On 23/03/2016 13:02, Russell Gold wrote:

:

Why does the module concept even need to exist at that point? Seems to me that 
it is much simpler to treat them as classes on the class path rather than a 
module. The module status can come in during packaging, no?

I'm not sure that I understand your mail. Are you asking why are modules 
are supported at compile-time? Or are you asking why the compiler 
doesn't allow packages to be split between the unnamed module and a 
named module?


-Alan.

Re: modulepath and classpath mixture

[Russell Gold]

> On Mar 23, 2016, at 3:21 AM, Alan Bateman  wrote:
> 
> On 22/03/2016 22:20, Russell Gold wrote:
>> I’d like to take a step back here. It may be that I have completely 
>> misunderstood what is going on, but this all seems to have gotten way more 
>> complicated than it should.
>> 
>> I am assuming:
>> 
>> 1) the project has both module and class path compile dependencies, and 
>> possibly has both module and class path test dependencies as well.
>> 2) the artifact type “module” (or something similar) is now recognized as 
>> indicating a Jigsaw module (which needs to go on the module path)
>> 3) the classes being built may be put into their own module, but might not 
>> be.
>> 
>> Then there are really four categories of dependencies. The compile step 
>> needs to place its jar dependencies on the class path and its module 
>> dependencies on the module path. I presume that is what Robert is already 
>> doing.
>> The test-compile step needs to handle all four, including the compiled class 
>> in its test class path.
>> 
>> Then there is no need to compile either the main or test classes as part of 
>> a module. They may use modules, but that is a different matter. No need for 
>> the -Xmodule switch at all. The only reference to module is as dependencies.
>> 
>> Did I completely miss the point?
>> 
> It depends on where the tests live. If they are black box tests that only 
> exercise the API (the public types in the module's exported packages) then 
> they can live anywhere. If they are in the same class loader and package as 
> the code they are testing (the norm in Maven) then they need to compiled as 
> if they are part of the module. We might think of these as white box tests as 
> they can make use of public types in packages that aren't exported, maybe 
> they are using package-private types/methods too.
> 

Why does the module concept even need to exist at that point? Seems to me that 
it is much simpler to treat them as classes on the class path rather than a 
module. The module status can come in during packaging, no?

- Russ

Re: modulepath and classpath mixture

[Stephen Colebourne]

On 23 March 2016 at 07:21, Alan Bateman  wrote:
> If they are in the same class loader and package as
> the code they are testing (the norm in Maven) then they need to compiled as
> if they are part of the module.

I struggle with that idea.

To me, tests are very definitely not part of the module. To me, they
are very clearly a separate module, one that uses the base module
(separate tree of code, separate dependencies, separate compilation
phase). In maven, there is even the possibility for one testing module
to depend on another testing module, a very useful feature.

The only special case about testing modules is that they are given
full access to everything in the underlying module. Forcing test code
to be embedded within the actual module just because the module design
doesn't allow any other option is flawed IMO.

Peter Levart's mail suggested one approach to provide the necessary
access rights for testing modules. But it still requires publication
package-by-package. Really, what is needed is a way to express that
the testing module is intimately connected, perhaps imposing a
requirement that both are loaded in the same classloader. eg.

module my.mod {
fully-exposed-to my.mod.test;
}
module my.mod.test {
requires my.mod;  // since other end specifies fully-exposed-to,
this gets full access
requires junit;
}

Is this issue on the list of open issues?
http://openjdk.java.net/projects/jigsaw/spec/issues/

Stephen

Upgrading EE modules (was Re: modulepath and classpath mixture)

[Alan Bateman]

On 22/03/2016 22:29, Richard Opalka wrote:

Hi,

   I'm experiencing the same problem locally.
When trying to build Oracle provided maven artifact from sources, namely

http://search.maven.org/#artifactdetails|javax.annotation|javax.annotation-api|1.2|jar 



the build fails on JDK9 (with Jigsaw) with message:

./src/javax/annotation/Priority.java:42: error: package exists in 
another module: java.annotations.common

package javax.annotation;
^
1 error

How is Jigsaw javac going to deal with such compilation scenarios?
I've changed the subject line as this issue seems to be about how to 
compile with, and make use of, the EE versions of the so-called "Common 
Annotations".


The JSR 250 defined Common Annotations is one of a small number of APIs 
that is shared between Java SE and Java EE. Java SE defines a subset, 
where Java EE defines the full API (or the full set of annotations in 
this case). With JDK 8 and older then Java EE or app servers needed to 
deploy an EE version to override the Java SE version. The only supported 
way of doing this was via the endorsed standards override mechanism 
(that mechanism was documented for both endorsed standards and 
standalone technologies with JSR 250 one of the standalone technologies).


As things currently stand, the replacement to the endorsed standards 
override mechanism is upgradeable modules. In JEP 261 you will see the 
option -upgrademodulepath that can be used to deploy modules that 
upgrade/override modules in the JDK.


So what you need here is the Java EE version of the 
java.annotations.common module and deploy it via -upgrademodulepath to 
upgrade/override the module in the JDK.


Unfortunately there isn't an EE version of this module yet so you have 
to create it yourself. Not hard, you just need to compile the following 
module-info.java and put it in the top-level directory of the JAR file 
to make it a modular JAR.


module java.annotations.common {
exports javax.annotation;
exports javax.annotation.security;
exports javax.annotation.sql;
}

Note that the Java SE/JDK version exports one package, the Java EE 
version exports three packages.


You can check that the EE version is observable with the following common:

  java -upgrademodulepath libs/javax.annotation-api.jar -listmods

or -listmods:java.annotations.common to have the module definition 
printed so that you can checks the exports.


-Alan

Re: modulepath and classpath mixture

[Alan Bateman]

On 22/03/2016 22:20, Russell Gold wrote:

I’d like to take a step back here. It may be that I have completely 
misunderstood what is going on, but this all seems to have gotten way more 
complicated than it should.

I am assuming:

1) the project has both module and class path compile dependencies, and 
possibly has both module and class path test dependencies as well.
2) the artifact type “module” (or something similar) is now recognized as 
indicating a Jigsaw module (which needs to go on the module path)
3) the classes being built may be put into their own module, but might not be.

Then there are really four categories of dependencies. The compile step needs 
to place its jar dependencies on the class path and its module dependencies on 
the module path. I presume that is what Robert is already doing.
The test-compile step needs to handle all four, including the compiled class in 
its test class path.

Then there is no need to compile either the main or test classes as part of a 
module. They may use modules, but that is a different matter. No need for the 
-Xmodule switch at all. The only reference to module is as dependencies.

Did I completely miss the point?

It depends on where the tests live. If they are black box tests that 
only exercise the API (the public types in the module's exported 
packages) then they can live anywhere. If they are in the same class 
loader and package as the code they are testing (the norm in Maven) then 
they need to compiled as if they are part of the module. We might think 
of these as white box tests as they can make use of public types in 
packages that aren't exported, maybe they are using package-private 
types/methods too.


-Alan

Re: modulepath and classpath mixture

[Alan Bateman]

On 22/03/2016 21:23, Robert Scholte wrote:
maven-builder-support/src/test/java/org/apache/maven/building/DefaultProblemTest.java:[1,1] 
package exists in another module: maven.builder.support


Just wondering, is this the expected behavior?

DefaultProblemTest is on the *classpath* and I wouldn't expect that these
entries would have any effect on the modulepath entries. I would 
expect that the packages between modules are checked, and that all 
these packages are exposed to the classpath; no extra check if 
classpath packages are in conflict with module packages.
The message says ' in *another* module', but this directory is not a 
module, which makes it extra confusing.


It looks like it is trying to compile 
org/apache/maven/building/DefaultProblemTest.java on the class path 
(unnamed module) but package org.apache.maven.building is in module 
maven.builder.support on the module path.


I think this is back to needing the module name as this can only work if 
you compile the test as if it is part of the module, meaning 
-Xmodule:maven.builder.support here.


-Alan

Re: modulepath and classpath mixture

[Richard Opalka]

Hi,

   I just solved it by adding -Xmodule option:

javac -Xmodule:java.annotations.common -sourcepath src `find -type f 
-name *.java`


Rio

On 03/22/2016 11:29 PM, Richard Opalka wrote:

Hi,

   I'm experiencing the same problem locally.
When trying to build Oracle provided maven artifact from sources, namely

http://search.maven.org/#artifactdetails|javax.annotation|javax.annotation-api|1.2|jar 



the build fails on JDK9 (with Jigsaw) with message:

./src/javax/annotation/Priority.java:42: error: package exists in 
another module: java.annotations.common

package javax.annotation;
^
1 error

How is Jigsaw javac going to deal with such compilation scenarios?

Rio

On 03/22/2016 10:23 PM, Robert Scholte wrote:
maven-builder-support/src/test/java/org/apache/maven/building/DefaultProblemTest.java:[1,1] 
package exists in another module: maven.builder.support


Just wondering, is this the expected behavior?

DefaultProblemTest is on the *classpath* and I wouldn't expect that 
these
entries would have any effect on the modulepath entries. I would 
expect that the packages between modules are checked, and that all 
these packages are exposed to the classpath; no extra check if 
classpath packages are in conflict with module packages.
The message says ' in *another* module', but this directory is not a 
module, which makes it extra confusing.


thanks,
Robert

On Thu, 17 Mar 2016 20:51:32 +0100, Robert Scholte 


wrote:


Hi,

it seems that simply adding -addmods ALL-MODULE-PATH isn't enough to 
compile the test-sources, and some already referred to it.


Here's the compiler error:
maven-builder-support/src/test/java/org/apache/maven/building/DefaultProblemTest.java:[1,1] 
package exists in another module: maven.builder.support


Let me quote Alan from one of the first responses on this thread:
"For the tests then I assume they are in the same packages as the 
sources under src/main/java, is that right?


In that case I think you will want to compile the tests as if they 
are part of the module:


   javac  -Xmodule:m  -d testclasses/m  -mp m.jar test/java/...

where m is the module name and the module (with sources in 
src/main/java) has already been compiled and then packaged as m.jar. 
The -Xmodule: option tells the compiler that you compiling 
the test classes as if they are part of module m. There is no 
module-info.java in the test tree."


To me it seems like the need for knowing the module name keeps 
returning.
This increases the need for a proper implementation of the 
maven-compiler-plugin as a multirelease JAR.
The pattern as shown during FOSDEM showed that the idea works, but 
it is not solid enough.
And the next question would be: can Maven (or actually Plexus 
ClassWorld) handle this?


I'll need to work out the things to be done and try to get more 
Maven developers involved.


thanks,
Robert

On Wed, 16 Mar 2016 21:55:01 +0100,  wrote:


2016/2/27 3:25 -0800, rfscho...@apache.org:

I noticed[1] that -addmods already has a special option: ALL-SYSTEM
What I'm looking for is something like ALL-MP or ALL-MODULEPATH, 
which
simply exposes all modules on the modulepath to the classpath. The 
set of
moduleEntries on the modulePath are already chosen with care and 
are in
the end all required to be able to compile the test-classes 
without the

need of knowing the name of the module being used to compile with.


We added a special ALL-MODULE-PATH token to the -addmods option, with
this description now in JEP 261 (http://openjdk.java.net/jeps/261):

  As a further special case, if `` is `ALL-MODULE-PATH` 
then all

  observable modules found on module paths are added to the root set.
  `ALL-MODULE-PATH` is valid at compile time when compiling classes 
in the
  unnamed module, and at run time when the main class of the 
application is

  loaded from the class path into the unnamed module.

This change is in Jigsaw EA build #4647 
(https://jdk9.java.net/jigsaw/).

Please try it out and let us know what you think.

- Mark

Re: modulepath and classpath mixture

[David M. Lloyd]

On 03/22/2016 07:06 AM, Peter Levart wrote:



On 02/24/2016 08:30 PM, Robert Scholte wrote:

On Wed, 24 Feb 2016 09:52:06 +0100, Alan Bateman
 wrote:



On 23/02/2016 21:10, Robert Scholte wrote:

:

If I understand this correctly I need to know the module name.

Has there been any discussion around having the module name in the
POM?  From the mails then it sounds like the project is mostly
"unaware" that it is creating a module. The other thing that comes to
mind is the source layout and whether it will get to the point where
the module name is in the file path. I'm mostly thinking of multi
module projects where one might have the source to multiple modules
in the same source tree.

-Alan


The name of the module with not end up in pom-4.0.0, it'll simply
break the xsd and would have effect on all other tools/IDEs/etc
depending on the pom.
The only place where one might specify the module name is in the
configuration of the maven-compiler-plugin.
In Brussels we talked about multimodules, but it makes more sense that
1 Maven project contains zero or one module-info[1].
And yes, I think a MavenProject will probably be unaware of its own
module name. It knows its source roots and outputdirectories (for both
main/java and test/java) and the packaged jar. Based on the
availability of the module-info in one of these locations it knows how
to construct the commandline arguments.
At least, this is what I'm hoping to achieve.

thanks,
Robert

[1] https://twitter.com/rfscholte/status/694599731515899904


Hi Robert,

Why do you want to support tests that "inject" classes into packages of
main artifact? Is this mainly because that's how it is done today? With
modules I would rather create tests in its own module and use qualified
exports from otherwise concealed packages of main module to give tests
access to types that are not accessible otherwise. This however means
that main module would have to use public qualifier on types and members
to allow test module to access them. In effect substituting the
package-private-nes with partly concealed packages to enable testability.

Perhaps jigsaw should realize that testabililty is an important aspect
to support. Forcing tools to compile a suitable set of javac and java
options to compile or inject new classes into an existing module and
augment it's dependencies is not very pleasant. Maybe this could be
supported with a little twist to the accessibility rules. Let me borrow
a rough idea that already circulated this list a while ago and modify it
to fit on the top of jigsaw accessibility rules.

Suppose that a qualified export of a package to the specific module(s)
could be augmented with say a "private" keyword:

module my.mod {
 requires ...;
 export private my.mod.internal to my.mod.test;
}

module my.mod.test {
 requires my.mod;
 requires junit;
}

...which would not only export the package to the specific module(s),
but also enable access to package-private types/members of that package
from the target module(s) as though those types/members were public for
them and for them only.


FWIW This is something I've been asking for in the JPMS EG.  I think 
this would be an incredibly powerful and useful security tool... and 
could also potentially eliminate lots of "shared secrets" from the JDK 
code base (and a few others as well).  Not just for tests but also for 
other existing code bases which today have to rely on 
privacy-by-convention/documentation for this kind of sharing.



It would only be possible to specify "private" keyword with qualified
exports - not with unqualified.

Would that be against any jigsaw goals?

Regards, Peter



--
- DML

Re: modulepath and classpath mixture

[Peter Levart]

On 02/24/2016 08:30 PM, Robert Scholte wrote:
On Wed, 24 Feb 2016 09:52:06 +0100, Alan Bateman 
 wrote:




On 23/02/2016 21:10, Robert Scholte wrote:

:

If I understand this correctly I need to know the module name.
Has there been any discussion around having the module name in the 
POM?  From the mails then it sounds like the project is mostly 
"unaware" that it is creating a module. The other thing that comes to 
mind is the source layout and whether it will get to the point where 
the module name is in the file path. I'm mostly thinking of multi 
module projects where one might have the source to multiple modules 
in the same source tree.


-Alan


The name of the module with not end up in pom-4.0.0, it'll simply 
break the xsd and would have effect on all other tools/IDEs/etc 
depending on the pom.
The only place where one might specify the module name is in the 
configuration of the maven-compiler-plugin.
In Brussels we talked about multimodules, but it makes more sense that 
1 Maven project contains zero or one module-info[1].
And yes, I think a MavenProject will probably be unaware of its own 
module name. It knows its source roots and outputdirectories (for both 
main/java and test/java) and the packaged jar. Based on the 
availability of the module-info in one of these locations it knows how 
to construct the commandline arguments.

At least, this is what I'm hoping to achieve.

thanks,
Robert

[1] https://twitter.com/rfscholte/status/694599731515899904


Hi Robert,

Why do you want to support tests that "inject" classes into packages of 
main artifact? Is this mainly because that's how it is done today? With 
modules I would rather create tests in its own module and use qualified 
exports from otherwise concealed packages of main module to give tests 
access to types that are not accessible otherwise. This however means 
that main module would have to use public qualifier on types and members 
to allow test module to access them. In effect substituting the 
package-private-nes with partly concealed packages to enable testability.


Perhaps jigsaw should realize that testabililty is an important aspect 
to support. Forcing tools to compile a suitable set of javac and java 
options to compile or inject new classes into an existing module and 
augment it's dependencies is not very pleasant. Maybe this could be 
supported with a little twist to the accessibility rules. Let me borrow 
a rough idea that already circulated this list a while ago and modify it 
to fit on the top of jigsaw accessibility rules.


Suppose that a qualified export of a package to the specific module(s) 
could be augmented with say a "private" keyword:


module my.mod {
requires ...;
export private my.mod.internal to my.mod.test;
}

module my.mod.test {
requires my.mod;
requires junit;
}

...which would not only export the package to the specific module(s), 
but also enable access to package-private types/members of that package 
from the target module(s) as though those types/members were public for 
them and for them only.


It would only be possible to specify "private" keyword with qualified 
exports - not with unqualified.


Would that be against any jigsaw goals?

Regards, Peter

Re: modulepath and classpath mixture

[Robert Scholte]

On Thu, 17 Mar 2016 21:23:25 +0100, Alan Bateman   
wrote:





On 17/03/2016 19:51, Robert Scholte wrote:

:

To me it seems like the need for knowing the module name keeps  
returning.
This increases the need for a proper implementation of the  
maven-compiler-plugin as a multirelease JAR.
The pattern as shown during FOSDEM showed that the idea works, but it  
is not solid enough.
And the next question would be: can Maven (or actually Plexus  
ClassWorld) handle this?


I'll need to work out the things to be done and try to get more Maven  
developers involved.
Would it you take it from the source module-info.class or the compiled  
module-info.class? The former would require a small parser. The latter  
is not difficult with ASM.


-Alan


I can do the former with QDox, for the latter I had JDK APIs in mind, but  
if ASM is an option too, that's an interesting option. Need to figure out  
how to do that.


Robert

Re: modulepath and classpath mixture

[Alan Bateman]

On 17/03/2016 19:51, Robert Scholte wrote:

:

To me it seems like the need for knowing the module name keeps returning.
This increases the need for a proper implementation of the 
maven-compiler-plugin as a multirelease JAR.
The pattern as shown during FOSDEM showed that the idea works, but it 
is not solid enough.
And the next question would be: can Maven (or actually Plexus 
ClassWorld) handle this?


I'll need to work out the things to be done and try to get more Maven 
developers involved.
Would it you take it from the source module-info.class or the compiled 
module-info.class? The former would require a small parser. The latter 
is not difficult with ASM.


-Alan

Re: modulepath and classpath mixture

[Paul Benedict]

I have an idea:

What if the jar tool in JDK 9 was upgraded so that the Manifest had a new
entry that contained the module name? This would allow access to it without
requiring new API, and tools running JDK 8 or below would have an easy way
to detect if the JAR is a module.

And is there a secondary benefit to decoupling the module name and the
module configuration? I'd like to hear another use case, if one exists.

On 08/03/2016 20:06, Paul Benedict wrote:
> >* Robert, it's sounds like a chicken-and-egg problem. You need the module
> *>* name to compile but don't know it until you have compiled.
> *>I think the scenario is where the module has been compiled and the issue
> is the separate compilation of the tests "as if" they are in the module.
>
> -Alan
>
>
Cheers,
Paul

Re: modulepath and classpath mixture

[Alan Bateman]

On 08/03/2016 20:06, Paul Benedict wrote:

Robert, it's sounds like a chicken-and-egg problem. You need the module
name to compile but don't know it until you have compiled.

I think the scenario is where the module has been compiled and the issue 
is the separate compilation of the tests "as if" they are in the module.


-Alan

Re: modulepath and classpath mixture

[Alan Bateman]

On 08/03/2016 19:13, Robert Scholte wrote:


This is how I thought that -Xpatch would work in short: the 
module-info in src/main/java and src/test/java have both the same 
modulename. The module-info in src/test/java specifies the extra 
required modules (like junit). With -Xpatch the test-classes have 
access to the non-exported main-classes too.
If the sources in src/test/java are a completely separate module then 
they will have their own module declaration.


If the tests are instead intended to augment the module, say where the 
tests are in the same package as the code under test, they will not have 
their own module declaration. This means no module-info.java in 
src/test/java. As things currently stand then javac -Xmodule:$MODULE 
will ignore the module-info.java when you compile the sources in 
src/test/java. At run-time then you'll get a warning if there is a 
module-info.class found when patching a module.


So I think you will need to get the module name when compiling the 
tests. You should not need the module name when compiling the code in 
src/main/java of course. The complication for the tests as they are 
being compiled "as if" they are part of the module.


As the tests classes in the same module as the main classes then it 
means the tests have access to all public types in the modules (no 
exports are needed). Furthermore, they are in the same package as the 
main classes then they have access to private package types/methods too.


The final part will be running the tests as the injected test classes 
will have dependences on JUnit or TestNG tests. They module declaration 
doesn't declare that it requires those and this is why we need to 
augment it at run-time with -XaddReads so that it reads the JUnit 
module. There are choices here as to whether JUnit is deployed on the 
class path or module path, it can work as either.


-Alan

Re: modulepath and classpath mixture

[Paul Benedict]

Bikeshed: JDK 9 is supposed to be feature complete 5/26/16 and be RC by
next January. Is this really enough time for EE Application Server projects
and other tools like Maven to vet the capabilities?

Cheers,
Paul

On Tue, Mar 8, 2016 at 4:08 PM, David M. Lloyd 
wrote:

> The only rational solution to this problem is to completely forego the
> JDK's capabilities and generate the file exclusively from the build
> tooling.  I expect that at some point we'll end up with a series of tools
> which construct the exports list from annotated package-infos, the requires
> list from a mapping or static list in the build, and a bunch of build- and
> application-system specific miscellany besides.  I don't believe that any
> nontrivial program is likely to bundle a module-info.java source file all,
> which makes me wonder if there's actually a point to having support for it
> at all.  Why write it when you can generate it, and at the same time, keep
> all your build-related information in one spot?
>
>
> On 03/08/2016 02:06 PM, Paul Benedict wrote:
>
>> Robert, it's sounds like a chicken-and-egg problem. You need the module
>> name to compile but don't know it until you have compiled.
>>
>> Too bad this file isn't XML so that any tool could read the module
>> information outside of compiling. That's what I advocated for a long time
>> but that battle has been lost.
>>
>>
>>
>> Cheers,
>> Paul
>>
>> On Tue, Mar 8, 2016 at 1:36 PM, Robert Scholte 
>> wrote:
>>
>> On Mon, 07 Mar 2016 14:53:28 +0100, Sander Mak 
>>> wrote:
>>>
>>>
>>> I don't think I understand the issue here. Using -Xpatch doesn't change

> the module declaration or export. It can be used to override or
> augment the
> module content, it just can't override the module declaration. It can
> be
> used in conjunction with -XaddReads and -XaddExports to read additional
> modules or export additional packages. For example, if a patch adds
> types
> to a new package then you could export that package with -XaddExports.
> If
> the patch injects tests into an existing package then those tests might
> have new dependences and requires compiling or running with
> -XaddReads:$MODULE=junit for example.
>
>
 I was playing around with exactly this yesterday, and this is what I
 ended up with:

 javac -Xmodule:javamodularity.easytext.algorithm.naivesyllablecounter \


 -XaddReads:javamodularity.easytext.algorithm.naivesyllablecounter=org.junit
 \
-mp mods:lib-test \
-d
 mods-test/javamodularity.easytext.algorithm.naivesyllablecounter
 $(find src-test -name '*.java')

 java -Xpatch:mods-test \


 -XaddReads:javamodularity.easytext.algorithm.naivesyllablecounter=org.junit
 \


 -XaddExports:javamodularity.easytext.algorithm.naivesyllablecounter/javamodularity.easytext.algorithm.naivesyllablecounter=org.junit
 \
   -mp mods:lib-test \
   -addmods
 javamodularity.easytext.algorithm.naivesyllablecounter,hamcrestcore \
   -m org.junit/org.junit.runner.JUnitCore

 javamodularity.easytext.algorithm.naivesyllablecounter.NaiveSyllableCounterTest

 Which patches my application module to contain a unit test, and then
 exposes my application module to junit at runtime (which is used as
 automatic module here). This works as expected.


 -- Sander


>>> When translating this to Maven it assumes that Maven is aware of the
>>> module name of the project is it building.
>>> Up until now that's not true. Developers specify the moduleName in the
>>> module-info.java and it doesn't make sense to ask them to add the same
>>> modulename to the pom (it that was possible) or the maven-compiler-plugin
>>> configuration. Instead Maven could use some new java9 APIs to discover
>>> the
>>> moduleName, but that would imply that from now on maven-compiler-plugin
>>> requires Java9 runtime. I don't think that's the way we want to go right
>>> now. Several Maven plugins have their own kind of multi-release pattern
>>> where some codeblocks depend on a specific Maven version, but we really
>>> want to avoid this.
>>> I hope we can find a way where Maven can simply refer to the
>>> classes-directory or the jar for some java/javac arguments where one
>>> would
>>> now need to be aware of its module name.
>>>  From Mavens point of view the output directories are facts, dependencies
>>> from the pom.xml too, as is the packaged artifact name & location, the
>>> content of java files are a mystery and not of any interest (at least in
>>> a
>>> classpath world ;) ).
>>>
>>> thanks,
>>> Robert
>>>
>>>
> --
> - DML
>

Re: modulepath and classpath mixture

[David M. Lloyd]

The only rational solution to this problem is to completely forego the 
JDK's capabilities and generate the file exclusively from the build 
tooling.  I expect that at some point we'll end up with a series of 
tools which construct the exports list from annotated package-infos, the 
requires list from a mapping or static list in the build, and a bunch of 
build- and application-system specific miscellany besides.  I don't 
believe that any nontrivial program is likely to bundle a 
module-info.java source file all, which makes me wonder if there's 
actually a point to having support for it at all.  Why write it when you 
can generate it, and at the same time, keep all your build-related 
information in one spot?


On 03/08/2016 02:06 PM, Paul Benedict wrote:

Robert, it's sounds like a chicken-and-egg problem. You need the module
name to compile but don't know it until you have compiled.

Too bad this file isn't XML so that any tool could read the module
information outside of compiling. That's what I advocated for a long time
but that battle has been lost.



Cheers,
Paul

On Tue, Mar 8, 2016 at 1:36 PM, Robert Scholte  wrote:


On Mon, 07 Mar 2016 14:53:28 +0100, Sander Mak 
wrote:



I don't think I understand the issue here. Using -Xpatch doesn't change

the module declaration or export. It can be used to override or augment the
module content, it just can't override the module declaration. It can be
used in conjunction with -XaddReads and -XaddExports to read additional
modules or export additional packages. For example, if a patch adds types
to a new package then you could export that package with -XaddExports. If
the patch injects tests into an existing package then those tests might
have new dependences and requires compiling or running with
-XaddReads:$MODULE=junit for example.



I was playing around with exactly this yesterday, and this is what I
ended up with:

javac -Xmodule:javamodularity.easytext.algorithm.naivesyllablecounter \

-XaddReads:javamodularity.easytext.algorithm.naivesyllablecounter=org.junit
\
   -mp mods:lib-test \
   -d mods-test/javamodularity.easytext.algorithm.naivesyllablecounter
$(find src-test -name '*.java')

java -Xpatch:mods-test \

  -XaddReads:javamodularity.easytext.algorithm.naivesyllablecounter=org.junit
\

  
-XaddExports:javamodularity.easytext.algorithm.naivesyllablecounter/javamodularity.easytext.algorithm.naivesyllablecounter=org.junit
\
  -mp mods:lib-test \
  -addmods
javamodularity.easytext.algorithm.naivesyllablecounter,hamcrestcore \
  -m org.junit/org.junit.runner.JUnitCore
javamodularity.easytext.algorithm.naivesyllablecounter.NaiveSyllableCounterTest

Which patches my application module to contain a unit test, and then
exposes my application module to junit at runtime (which is used as
automatic module here). This works as expected.


-- Sander



When translating this to Maven it assumes that Maven is aware of the
module name of the project is it building.
Up until now that's not true. Developers specify the moduleName in the
module-info.java and it doesn't make sense to ask them to add the same
modulename to the pom (it that was possible) or the maven-compiler-plugin
configuration. Instead Maven could use some new java9 APIs to discover the
moduleName, but that would imply that from now on maven-compiler-plugin
requires Java9 runtime. I don't think that's the way we want to go right
now. Several Maven plugins have their own kind of multi-release pattern
where some codeblocks depend on a specific Maven version, but we really
want to avoid this.
I hope we can find a way where Maven can simply refer to the
classes-directory or the jar for some java/javac arguments where one would
now need to be aware of its module name.
 From Mavens point of view the output directories are facts, dependencies
from the pom.xml too, as is the packaged artifact name & location, the
content of java files are a mystery and not of any interest (at least in a
classpath world ;) ).

thanks,
Robert



--
- DML

Re: modulepath and classpath mixture

[Paul Benedict]

Robert, it's sounds like a chicken-and-egg problem. You need the module
name to compile but don't know it until you have compiled.

Too bad this file isn't XML so that any tool could read the module
information outside of compiling. That's what I advocated for a long time
but that battle has been lost.



Cheers,
Paul

On Tue, Mar 8, 2016 at 1:36 PM, Robert Scholte  wrote:

> On Mon, 07 Mar 2016 14:53:28 +0100, Sander Mak 
> wrote:
>
>
>> I don't think I understand the issue here. Using -Xpatch doesn't change
>>> the module declaration or export. It can be used to override or augment the
>>> module content, it just can't override the module declaration. It can be
>>> used in conjunction with -XaddReads and -XaddExports to read additional
>>> modules or export additional packages. For example, if a patch adds types
>>> to a new package then you could export that package with -XaddExports. If
>>> the patch injects tests into an existing package then those tests might
>>> have new dependences and requires compiling or running with
>>> -XaddReads:$MODULE=junit for example.
>>>
>>
>> I was playing around with exactly this yesterday, and this is what I
>> ended up with:
>>
>> javac -Xmodule:javamodularity.easytext.algorithm.naivesyllablecounter \
>>
>> -XaddReads:javamodularity.easytext.algorithm.naivesyllablecounter=org.junit
>> \
>>   -mp mods:lib-test \
>>   -d mods-test/javamodularity.easytext.algorithm.naivesyllablecounter
>> $(find src-test -name '*.java')
>>
>> java -Xpatch:mods-test \
>>
>>  -XaddReads:javamodularity.easytext.algorithm.naivesyllablecounter=org.junit
>> \
>>
>>  
>> -XaddExports:javamodularity.easytext.algorithm.naivesyllablecounter/javamodularity.easytext.algorithm.naivesyllablecounter=org.junit
>> \
>>  -mp mods:lib-test \
>>  -addmods
>> javamodularity.easytext.algorithm.naivesyllablecounter,hamcrestcore \
>>  -m org.junit/org.junit.runner.JUnitCore
>> javamodularity.easytext.algorithm.naivesyllablecounter.NaiveSyllableCounterTest
>>
>> Which patches my application module to contain a unit test, and then
>> exposes my application module to junit at runtime (which is used as
>> automatic module here). This works as expected.
>>
>>
>> -- Sander
>>
>
> When translating this to Maven it assumes that Maven is aware of the
> module name of the project is it building.
> Up until now that's not true. Developers specify the moduleName in the
> module-info.java and it doesn't make sense to ask them to add the same
> modulename to the pom (it that was possible) or the maven-compiler-plugin
> configuration. Instead Maven could use some new java9 APIs to discover the
> moduleName, but that would imply that from now on maven-compiler-plugin
> requires Java9 runtime. I don't think that's the way we want to go right
> now. Several Maven plugins have their own kind of multi-release pattern
> where some codeblocks depend on a specific Maven version, but we really
> want to avoid this.
> I hope we can find a way where Maven can simply refer to the
> classes-directory or the jar for some java/javac arguments where one would
> now need to be aware of its module name.
> From Mavens point of view the output directories are facts, dependencies
> from the pom.xml too, as is the packaged artifact name & location, the
> content of java files are a mystery and not of any interest (at least in a
> classpath world ;) ).
>
> thanks,
> Robert
>

Re: modulepath and classpath mixture

[Robert Scholte]

On Mon, 07 Mar 2016 14:53:28 +0100, Sander Mak   
wrote:




I don't think I understand the issue here. Using -Xpatch doesn't change  
the module declaration or export. It can be used to override or augment  
the module content, it just can't override the module declaration. It  
can be used in conjunction with -XaddReads and -XaddExports to read  
additional modules or export additional packages. For example, if a  
patch adds types to a new package then you could export that package  
with -XaddExports. If the patch injects tests into an existing package  
then those tests might have new dependences and requires compiling or  
running with -XaddReads:$MODULE=junit for example.


I was playing around with exactly this yesterday, and this is what I  
ended up with:


javac -Xmodule:javamodularity.easytext.algorithm.naivesyllablecounter \
  -XaddReads:javamodularity.easytext.algorithm.naivesyllablecounter=org.junit  
\

  -mp mods:lib-test \
  -d  
mods-test/javamodularity.easytext.algorithm.naivesyllablecounter $(find  
src-test -name '*.java')


java -Xpatch:mods-test \
 -XaddReads:javamodularity.easytext.algorithm.naivesyllablecounter=org.junit  
\
 -XaddExports:javamodularity.easytext.algorithm.naivesyllablecounter/javamodularity.easytext.algorithm.naivesyllablecounter=org.junit  
\

 -mp mods:lib-test \
 -addmods  
javamodularity.easytext.algorithm.naivesyllablecounter,hamcrestcore \
 -m org.junit/org.junit.runner.JUnitCore  
javamodularity.easytext.algorithm.naivesyllablecounter.NaiveSyllableCounterTest


Which patches my application module to contain a unit test, and then  
exposes my application module to junit at runtime (which is used as  
automatic module here). This works as expected.



-- Sander


When translating this to Maven it assumes that Maven is aware of the  
module name of the project is it building.
Up until now that's not true. Developers specify the moduleName in the  
module-info.java and it doesn't make sense to ask them to add the same  
modulename to the pom (it that was possible) or the maven-compiler-plugin  
configuration. Instead Maven could use some new java9 APIs to discover the  
moduleName, but that would imply that from now on maven-compiler-plugin  
requires Java9 runtime. I don't think that's the way we want to go right  
now. Several Maven plugins have their own kind of multi-release pattern  
where some codeblocks depend on a specific Maven version, but we really  
want to avoid this.
I hope we can find a way where Maven can simply refer to the  
classes-directory or the jar for some java/javac arguments where one would  
now need to be aware of its module name.
From Mavens point of view the output directories are facts, dependencies  
from the pom.xml too, as is the packaged artifact name & location, the  
content of java files are a mystery and not of any interest (at least in a  
classpath world ;) ).


thanks,
Robert

Re: modulepath and classpath mixture

[Robert Scholte]

On Mon, 07 Mar 2016 13:13:38 +0100, Alan Bateman   
wrote:



On 06/03/2016 14:01, Robert Scholte wrote:

Hi,

I've asked for some feedback and there seems to be concerns with split  
packages when there are two or more modules on the module path that  
export the same package.
Unless I misunderstand the issue, I'd say you have the same problem  
with -addmods. If you add mod1 and mod2, which both export the same  
package, you have exactly the same issue, right?
Yes, although at least if you specify the module names to -addmods then  
you are being explicit as to the additional modules to resolve. The  
issue with -addmods ALL-NAMED (or what the token is) is that it will  
resolve all modules that can be found via the application module path  
and so would need to be used with great care.




I can only speak for Maven how it wants to use it. Only the modules used  
to compile the src/main/java sources will end up on the modulePath in this  
case. So the set of modules has already been validated, kind of.




When talking about exports it made me realize there's probably another  
issue: only the exported packages of the "main"-module are accessible,  
right? Since src/test/java has no module-info, the -Xpatch is useless.
An idea that comes to my mind is something like -mainModule ,  
where are either a jar or directory containing module-info. All its  
classes can be accessed by the classes on the classpath, all other  
modules keep their package access restrictions.
I don't think I understand the issue here. Using -Xpatch doesn't change  
the module declaration or export. It can be used to override or augment  
the module content, it just can't override the module declaration. It  
can be used in conjunction with -XaddReads and -XaddExports to read  
additional modules or export additional packages. For example, if a  
patch adds types to a new package then you could export that package  
with -XaddExports. If the patch injects tests into an existing package  
then those tests might have new dependences and requires compiling or  
running with -XaddReads:$MODULE=junit for example.


This is how I thought that -Xpatch would work in short: the module-info in  
src/main/java and src/test/java have both the same modulename. The  
module-info in src/test/java specifies the extra required modules (like  
junit). With -Xpatch the test-classes have access to the non-exported  
main-classes too.




-Alan


thanks,
Robert

Re: modulepath and classpath mixture

[Sander Mak]

> I don't think I understand the issue here. Using -Xpatch doesn't change the 
> module declaration or export. It can be used to override or augment the 
> module content, it just can't override the module declaration. It can be used 
> in conjunction with -XaddReads and -XaddExports to read additional modules or 
> export additional packages. For example, if a patch adds types to a new 
> package then you could export that package with -XaddExports. If the patch 
> injects tests into an existing package then those tests might have new 
> dependences and requires compiling or running with -XaddReads:$MODULE=junit 
> for example.

I was playing around with exactly this yesterday, and this is what I ended up 
with:

javac -Xmodule:javamodularity.easytext.algorithm.naivesyllablecounter \
  
-XaddReads:javamodularity.easytext.algorithm.naivesyllablecounter=org.junit \
  -mp mods:lib-test \
  -d mods-test/javamodularity.easytext.algorithm.naivesyllablecounter 
$(find src-test -name '*.java')

java -Xpatch:mods-test \
 
-XaddReads:javamodularity.easytext.algorithm.naivesyllablecounter=org.junit \
 
-XaddExports:javamodularity.easytext.algorithm.naivesyllablecounter/javamodularity.easytext.algorithm.naivesyllablecounter=org.junit
 \
 -mp mods:lib-test \
 -addmods 
javamodularity.easytext.algorithm.naivesyllablecounter,hamcrestcore \
 -m org.junit/org.junit.runner.JUnitCore 
javamodularity.easytext.algorithm.naivesyllablecounter.NaiveSyllableCounterTest

Which patches my application module to contain a unit test, and then exposes my 
application module to junit at runtime (which is used as automatic module 
here). This works as expected.


-- Sander

Re: modulepath and classpath mixture

[Alan Bateman]

On 06/03/2016 14:01, Robert Scholte wrote:

Hi,

I've asked for some feedback and there seems to be concerns with split 
packages when there are two or more modules on the module path that 
export the same package.
Unless I misunderstand the issue, I'd say you have the same problem 
with -addmods. If you add mod1 and mod2, which both export the same 
package, you have exactly the same issue, right?
Yes, although at least if you specify the module names to -addmods then 
you are being explicit as to the additional modules to resolve. The 
issue with -addmods ALL-NAMED (or what the token is) is that it will 
resolve all modules that can be found via the application module path 
and so would need to be used with great care.




When talking about exports it made me realize there's probably another 
issue: only the exported packages of the "main"-module are accessible, 
right? Since src/test/java has no module-info, the -Xpatch is useless.
An idea that comes to my mind is something like -mainModule , 
where are either a jar or directory containing module-info. All its 
classes can be accessed by the classes on the classpath, all other 
modules keep their package access restrictions.
I don't think I understand the issue here. Using -Xpatch doesn't change 
the module declaration or export. It can be used to override or augment 
the module content, it just can't override the module declaration. It 
can be used in conjunction with -XaddReads and -XaddExports to read 
additional modules or export additional packages. For example, if a 
patch adds types to a new package then you could export that package 
with -XaddExports. If the patch injects tests into an existing package 
then those tests might have new dependences and requires compiling or 
running with -XaddReads:$MODULE=junit for example.


-Alan

Re: modulepath and classpath mixture

[Jonathan Gibbons]

At the risk of opening bikesheds, if we go that way, I would suggest 
just  -ALL-   or just a new option -addallmods.


-- Jon


On 02/27/2016 03:25 AM, Robert Scholte wrote:

Hi,

I noticed[1] that -addmods already has a special option: ALL-SYSTEM
What I'm looking for is something like ALL-MP or ALL-MODULEPATH, which 
simply exposes all modules on the modulepath to the classpath. The set 
of moduleEntries on the modulePath are already chosen with care and 
are in the end all required to be able to compile the test-classes 
without the need of knowing the name of the module being used to 
compile with.


thanks,
Robert

[1] http://openjdk.java.net/jeps/261


On Tue, 23 Feb 2016 01:52:50 +0100, Jonathan Gibbons 
 wrote:





On 02/22/2016 12:44 PM, Robert Scholte wrote:

Hi,

first of all I'd like to say that I'm very pleased with the new -mp 
options, these matches better with the way Apache Maven would like 
to work with jars and class-folders.


Here's my use case: I noticed that if I add a module-info to 
src/main/java and put all compile-scoped dependencies to the module 
path, all compiles fines.
I assume that developers are less interested in adding a 
module-info.java file to src/test/java, so that's what I'm doing 
right now too.
Now it seems that I *must* add compile + test scoped to the 
*classpath* to be able to compile the test classes.
My first approach was to leave the compile-scoped dependencies on 
the modulepath and all test-scoped dependencies on the classpath, so 
the modules keeps their inner related structure, but it seems that 
the classpath classes cannot access the modulepath classes.


I'm looking for the confirmation that putting all dependencies on 
the classpath is indeed the right approach in this case.


thanks,
Robert


Robert,

We definitely need some more detailed notes on setting up javac 
compilations (note to self!) but one thing to note is that by 
default, the unnamed module (i.e. code on the classpath) only has 
observability of the modules in the system image. To make modules on 
the module path observable, you need to use the -addmods option.


-- Jon

Re: modulepath and classpath mixture

[Robert Scholte]

Hi,

I noticed[1] that -addmods already has a special option: ALL-SYSTEM
What I'm looking for is something like ALL-MP or ALL-MODULEPATH, which  
simply exposes all modules on the modulepath to the classpath. The set of  
moduleEntries on the modulePath are already chosen with care and are in  
the end all required to be able to compile the test-classes without the  
need of knowing the name of the module being used to compile with.


thanks,
Robert

[1] http://openjdk.java.net/jeps/261


On Tue, 23 Feb 2016 01:52:50 +0100, Jonathan Gibbons  
 wrote:





On 02/22/2016 12:44 PM, Robert Scholte wrote:

Hi,

first of all I'd like to say that I'm very pleased with the new -mp  
options, these matches better with the way Apache Maven would like to  
work with jars and class-folders.


Here's my use case: I noticed that if I add a module-info to  
src/main/java and put all compile-scoped dependencies to the module  
path, all compiles fines.
I assume that developers are less interested in adding a  
module-info.java file to src/test/java, so that's what I'm doing right  
now too.
Now it seems that I *must* add compile + test scoped to the *classpath*  
to be able to compile the test classes.
My first approach was to leave the compile-scoped dependencies on the  
modulepath and all test-scoped dependencies on the classpath, so the  
modules keeps their inner related structure, but it seems that the  
classpath classes cannot access the modulepath classes.


I'm looking for the confirmation that putting all dependencies on the  
classpath is indeed the right approach in this case.


thanks,
Robert


Robert,

We definitely need some more detailed notes on setting up javac  
compilations (note to self!) but one thing to note is that by default,  
the unnamed module (i.e. code on the classpath) only has observability  
of the modules in the system image. To make modules on the module path  
observable, you need to use the -addmods option.


-- Jon



--
Using Opera's mail client: http://www.opera.com/mail/

Re: modulepath and classpath mixture

[Alan Bateman]

On 24/02/2016 19:12, Robert Scholte wrote:


Hmm, would have been nice if I had known about these discussions, 
because I don't think that this is a valid assumption from a Maven 
perspective. Ideally developers simply add module-info.java files to 
the source-roots of their choice and Maven should be able to construct 
the correct set of javac arguments.
I don't expect developers to open a jar to see if there's a 
module-info available. Actually, how can he figure out the module 
name, since the module-info is a compiled file?
One way is `jar --file foo.jar -p`. That will print the module 
descriptor when the JAR file is a modular JAR. There is API support for 
reading the binary form of the module declaration too.


-Alan

Re: modulepath and classpath mixture

[Robert Scholte]

On Wed, 24 Feb 2016 09:52:06 +0100, Alan Bateman   
wrote:




On 23/02/2016 21:10, Robert Scholte wrote:

:

If I understand this correctly I need to know the module name.
Has there been any discussion around having the module name in the POM?  
 From the mails then it sounds like the project is mostly "unaware" that  
it is creating a module. The other thing that comes to mind is the  
source layout and whether it will get to the point where the module name  
is in the file path. I'm mostly thinking of multi module projects where  
one might have the source to multiple modules in the same source tree.


-Alan


The name of the module with not end up in pom-4.0.0, it'll simply break  
the xsd and would have effect on all other tools/IDEs/etc depending on the  
pom.
The only place where one might specify the module name is in the  
configuration of the maven-compiler-plugin.
In Brussels we talked about multimodules, but it makes more sense that 1  
Maven project contains zero or one module-info[1].
And yes, I think a MavenProject will probably be unaware of its own module  
name. It knows its source roots and outputdirectories (for both main/java  
and test/java) and the packaged jar. Based on the availability of the  
module-info in one of these locations it knows how to construct the  
commandline arguments.

At least, this is what I'm hoping to achieve.

thanks,
Robert

[1] https://twitter.com/rfscholte/status/694599731515899904

Re: modulepath and classpath mixture

[Robert Scholte]

On Wed, 24 Feb 2016 00:15:26 +0100, Jonathan Gibbons  
 wrote:





On 02/23/2016 01:22 PM, Robert Scholte wrote:
On Tue, 23 Feb 2016 22:14:32 +0100, Jonathan Gibbons  
 wrote:





On 02/23/2016 01:06 PM, Jonathan Gibbons wrote:



On 02/23/2016 12:48 PM, Robert Scholte wrote:
On Tue, 23 Feb 2016 01:52:50 +0100, Jonathan Gibbons  
 wrote:





On 02/22/2016 12:44 PM, Robert Scholte wrote:

Hi,

first of all I'd like to say that I'm very pleased with the new  
-mp options, these matches better with the way Apache Maven would  
like to work with jars and class-folders.


Here's my use case: I noticed that if I add a module-info to  
src/main/java and put all compile-scoped dependencies to the  
module path, all compiles fines.
I assume that developers are less interested in adding a  
module-info.java file to src/test/java, so that's what I'm doing  
right now too.
Now it seems that I *must* add compile + test scoped to the  
*classpath* to be able to compile the test classes.
My first approach was to leave the compile-scoped dependencies on  
the modulepath and all test-scoped dependencies on the classpath,  
so the modules keeps their inner related structure, but it seems  
that the classpath classes cannot access the modulepath classes.


I'm looking for the confirmation that putting all dependencies on  
the classpath is indeed the right approach in this case.


thanks,
Robert


Robert,

We definitely need some more detailed notes on setting up javac  
compilations (note to self!) but one thing to note is that by  
default, the unnamed module (i.e. code on the classpath) only has  
observability of the modules in the system image. To make modules  
on the module path observable, you need to use the -addmods option.


-- Jon


Hi Jonathan,

this would indeed explain what I'm facing right now. However, adding  
-addmods gives me the following exception:
Caused by: java.lang.IllegalArgumentException: -addmods requires an  
argument
at  
com.sun.tools.javac.main.Arguments.error(jdk.compiler@9-ea/Arguments.java:708)


Is -addmods followed by the same entries as -modulepath or by the  
modulenames. I really hope it is not the latter, because that would  
mean that I first need to discover and read all module-info files.


thanks,
Robert


Sorry, I should have been more explicit.

Both javac and java (the launcher) accept an option "-addmods  
,..." which can be used to name modules to be included  
in the module graph.   Confusingly, for javac, the option is listed  
under javac -X (that's a bug we will fix), but setting that aside,  
here's what the command line help says:


  -addmods [,...] Root modules to resolve in  
addition to the initial modules


"java -help" says effectively the same.


So yes, the option takes a list of module names, not module paths.

-- Jon







... but that being said, note that you don't have to list all the  
modules on the module path.You only need to list root modules, and  
javac will determine the transitive closure of all the necessary  
modules.


So, if you're writing tests in the unnamed module, to test a module M,  
the chances are that you only need "-addmods M".




This makes sense. And although I still don't like the fact that this  
requires me to read the module-info, this should be possible for the  
target/mods/m (since it is already compiled).

So my response to Alan was probably a bit too fast.
This requires some tricks on our side to stay compatible with lower  
Java versions while adding some code to read the module-info.


thanks,
Robert



When we've had discussions about how these options might work, we've  
generally assumed you might have some a priori knowledge of the module  
name from some other context, rather than having to rely on reading  
module info.


-- Jon



Hmm, would have been nice if I had known about these discussions, because  
I don't think that this is a valid assumption from a Maven perspective.  
Ideally developers simply add module-info.java files to the source-roots  
of their choice and Maven should be able to construct the correct set of  
javac arguments.
I don't expect developers to open a jar to see if there's a module-info  
available. Actually, how can he figure out the module name, since the  
module-info is a compiled file?
Anyhow, Maven is capable to discover the module name when required, but it  
is not that efficient. Maybe it is time to work on some feedback to  
describe the issues I'm facing regarding some of the javac options.


thanks,
Robert






Alan wrote a separate email about different compilation scenarios.  
Note that in many/most of those cases, no -addmods was necessary.



-- Jon








--
Using Opera's mail client: http://www.opera.com/mail/

Re: modulepath and classpath mixture

[Alan Bateman]

On 23/02/2016 21:10, Robert Scholte wrote:

:

If I understand this correctly I need to know the module name.
Has there been any discussion around having the module name in the POM? 
From the mails then it sounds like the project is mostly "unaware" that 
it is creating a module. The other thing that comes to mind is the 
source layout and whether it will get to the point where the module name 
is in the file path. I'm mostly thinking of multi module projects where 
one might have the source to multiple modules in the same source tree.


-Alan

Re: modulepath and classpath mixture

[Jonathan Gibbons]

On 02/23/2016 01:22 PM, Robert Scholte wrote:
On Tue, 23 Feb 2016 22:14:32 +0100, Jonathan Gibbons 
 wrote:





On 02/23/2016 01:06 PM, Jonathan Gibbons wrote:



On 02/23/2016 12:48 PM, Robert Scholte wrote:
On Tue, 23 Feb 2016 01:52:50 +0100, Jonathan Gibbons 
 wrote:





On 02/22/2016 12:44 PM, Robert Scholte wrote:

Hi,

first of all I'd like to say that I'm very pleased with the new 
-mp options, these matches better with the way Apache Maven would 
like to work with jars and class-folders.


Here's my use case: I noticed that if I add a module-info to 
src/main/java and put all compile-scoped dependencies to the 
module path, all compiles fines.
I assume that developers are less interested in adding a 
module-info.java file to src/test/java, so that's what I'm doing 
right now too.
Now it seems that I *must* add compile + test scoped to the 
*classpath* to be able to compile the test classes.
My first approach was to leave the compile-scoped dependencies on 
the modulepath and all test-scoped dependencies on the classpath, 
so the modules keeps their inner related structure, but it seems 
that the classpath classes cannot access the modulepath classes.


I'm looking for the confirmation that putting all dependencies on 
the classpath is indeed the right approach in this case.


thanks,
Robert


Robert,

We definitely need some more detailed notes on setting up javac 
compilations (note to self!) but one thing to note is that by 
default, the unnamed module (i.e. code on the classpath) only has 
observability of the modules in the system image. To make modules 
on the module path observable, you need to use the -addmods option.


-- Jon


Hi Jonathan,

this would indeed explain what I'm facing right now. However, 
adding -addmods gives me the following exception:
Caused by: java.lang.IllegalArgumentException: -addmods requires an 
argument
at 
com.sun.tools.javac.main.Arguments.error(jdk.compiler@9-ea/Arguments.java:708)


Is -addmods followed by the same entries as -modulepath or by the 
modulenames. I really hope it is not the latter, because that would 
mean that I first need to discover and read all module-info files.


thanks,
Robert


Sorry, I should have been more explicit.

Both javac and java (the launcher) accept an option "-addmods 
,..." which can be used to name modules to be included 
in the module graph.   Confusingly, for javac, the option is listed 
under javac -X (that's a bug we will fix), but setting that aside, 
here's what the command line help says:


  -addmods [,...] Root modules to resolve in 
addition to the initial modules


"java -help" says effectively the same.


So yes, the option takes a list of module names, not module paths.

-- Jon







... but that being said, note that you don't have to list all the 
modules on the module path.You only need to list root modules, 
and javac will determine the transitive closure of all the necessary 
modules.


So, if you're writing tests in the unnamed module, to test a module 
M, the chances are that you only need "-addmods M".




This makes sense. And although I still don't like the fact that this 
requires me to read the module-info, this should be possible for the 
target/mods/m (since it is already compiled).

So my response to Alan was probably a bit too fast.
This requires some tricks on our side to stay compatible with lower 
Java versions while adding some code to read the module-info.


thanks,
Robert



When we've had discussions about how these options might work, we've 
generally assumed you might have some a priori knowledge of the module 
name from some other context, rather than having to rely on reading 
module info.


-- Jon






Alan wrote a separate email about different compilation scenarios. 
Note that in many/most of those cases, no -addmods was necessary.



-- Jon

Re: modulepath and classpath mixture

[Robert Scholte]

On Tue, 23 Feb 2016 22:14:32 +0100, Jonathan Gibbons  
 wrote:





On 02/23/2016 01:06 PM, Jonathan Gibbons wrote:



On 02/23/2016 12:48 PM, Robert Scholte wrote:
On Tue, 23 Feb 2016 01:52:50 +0100, Jonathan Gibbons  
 wrote:





On 02/22/2016 12:44 PM, Robert Scholte wrote:

Hi,

first of all I'd like to say that I'm very pleased with the new -mp  
options, these matches better with the way Apache Maven would like  
to work with jars and class-folders.


Here's my use case: I noticed that if I add a module-info to  
src/main/java and put all compile-scoped dependencies to the module  
path, all compiles fines.
I assume that developers are less interested in adding a  
module-info.java file to src/test/java, so that's what I'm doing  
right now too.
Now it seems that I *must* add compile + test scoped to the  
*classpath* to be able to compile the test classes.
My first approach was to leave the compile-scoped dependencies on  
the modulepath and all test-scoped dependencies on the classpath, so  
the modules keeps their inner related structure, but it seems that  
the classpath classes cannot access the modulepath classes.


I'm looking for the confirmation that putting all dependencies on  
the classpath is indeed the right approach in this case.


thanks,
Robert


Robert,

We definitely need some more detailed notes on setting up javac  
compilations (note to self!) but one thing to note is that by  
default, the unnamed module (i.e. code on the classpath) only has  
observability of the modules in the system image. To make modules on  
the module path observable, you need to use the -addmods option.


-- Jon


Hi Jonathan,

this would indeed explain what I'm facing right now. However, adding  
-addmods gives me the following exception:
Caused by: java.lang.IllegalArgumentException: -addmods requires an  
argument
at  
com.sun.tools.javac.main.Arguments.error(jdk.compiler@9-ea/Arguments.java:708)


Is -addmods followed by the same entries as -modulepath or by the  
modulenames. I really hope it is not the latter, because that would  
mean that I first need to discover and read all module-info files.


thanks,
Robert


Sorry, I should have been more explicit.

Both javac and java (the launcher) accept an option "-addmods  
,..." which can be used to name modules to be included in  
the module graph.   Confusingly, for javac, the option is listed under  
javac -X (that's a bug we will fix), but setting that aside, here's  
what the command line help says:


  -addmods [,...] Root modules to resolve in  
addition to the initial modules


"java -help" says effectively the same.


So yes, the option takes a list of module names, not module paths.

-- Jon







... but that being said, note that you don't have to list all the  
modules on the module path.You only need to list root modules, and  
javac will determine the transitive closure of all the necessary modules.


So, if you're writing tests in the unnamed module, to test a module M,  
the chances are that you only need "-addmods M".




This makes sense. And although I still don't like the fact that this  
requires me to read the module-info, this should be possible for the  
target/mods/m (since it is already compiled).

So my response to Alan was probably a bit too fast.
This requires some tricks on our side to stay compatible with lower Java  
versions while adding some code to read the module-info.


thanks,
Robert




Alan wrote a separate email about different compilation scenarios. Note  
that in many/most of those cases, no -addmods was necessary.



-- Jon



--
Using Opera's mail client: http://www.opera.com/mail/

Re: modulepath and classpath mixture

[Jonathan Gibbons]

On 02/23/2016 01:10 PM, Robert Scholte wrote:


And maybe this is the key question: if src/main/java is a module, 
should we handle src/test/java as a module too or leave it as a 
classpath based project?


thanks,
Robert 



You list 2 choices, but there's 3 possible answers here.

If you're writing a test that only exercises the public/exported API of 
the module, it could either be written as code in the unnamed module (a 
"classpath based project"), or it could be written as a separate, named 
module.


If your test is using the pattern of putting test classes in the same 
package as that which is being tested, then the test has to be part of 
the module being tested, which means you get into using -Xmodule at 
compile time and -Xpatch at runtime.


-- Jon

Re: modulepath and classpath mixture

[Jonathan Gibbons]

On 02/23/2016 01:06 PM, Jonathan Gibbons wrote:



On 02/23/2016 12:48 PM, Robert Scholte wrote:
On Tue, 23 Feb 2016 01:52:50 +0100, Jonathan Gibbons 
 wrote:





On 02/22/2016 12:44 PM, Robert Scholte wrote:

Hi,

first of all I'd like to say that I'm very pleased with the new -mp 
options, these matches better with the way Apache Maven would like 
to work with jars and class-folders.


Here's my use case: I noticed that if I add a module-info to 
src/main/java and put all compile-scoped dependencies to the module 
path, all compiles fines.
I assume that developers are less interested in adding a 
module-info.java file to src/test/java, so that's what I'm doing 
right now too.
Now it seems that I *must* add compile + test scoped to the 
*classpath* to be able to compile the test classes.
My first approach was to leave the compile-scoped dependencies on 
the modulepath and all test-scoped dependencies on the classpath, 
so the modules keeps their inner related structure, but it seems 
that the classpath classes cannot access the modulepath classes.


I'm looking for the confirmation that putting all dependencies on 
the classpath is indeed the right approach in this case.


thanks,
Robert


Robert,

We definitely need some more detailed notes on setting up javac 
compilations (note to self!) but one thing to note is that by 
default, the unnamed module (i.e. code on the classpath) only has 
observability of the modules in the system image. To make modules on 
the module path observable, you need to use the -addmods option.


-- Jon


Hi Jonathan,

this would indeed explain what I'm facing right now. However, adding 
-addmods gives me the following exception:
Caused by: java.lang.IllegalArgumentException: -addmods requires an 
argument
at 
com.sun.tools.javac.main.Arguments.error(jdk.compiler@9-ea/Arguments.java:708)


Is -addmods followed by the same entries as -modulepath or by the 
modulenames. I really hope it is not the latter, because that would 
mean that I first need to discover and read all module-info files.


thanks,
Robert


Sorry, I should have been more explicit.

Both javac and java (the launcher) accept an option "-addmods 
,..." which can be used to name modules to be included in 
the module graph.   Confusingly, for javac, the option is listed under 
javac -X (that's a bug we will fix), but setting that aside, here's 
what the command line help says:


  -addmods [,...] Root modules to resolve in 
addition to the initial modules


"java -help" says effectively the same.


So yes, the option takes a list of module names, not module paths.

-- Jon







... but that being said, note that you don't have to list all the 
modules on the module path.You only need to list root modules, and 
javac will determine the transitive closure of all the necessary modules.


So, if you're writing tests in the unnamed module, to test a module M, 
the chances are that you only need "-addmods M".



Alan wrote a separate email about different compilation scenarios. Note 
that in many/most of those cases, no -addmods was necessary.



-- Jon

Re: modulepath and classpath mixture

[Robert Scholte]

On Tue, 23 Feb 2016 01:30:16 +0100, Alex Buckley   
wrote:



Hi Robert,

On 2/22/2016 12:44 PM, Robert Scholte wrote:

Here's my use case: I noticed that if I add a module-info to
src/main/java and put all compile-scoped dependencies to the module
path, all compiles fines.


Sounds good.


I assume that developers are less interested in adding a
module-info.java file to src/test/java, so that's what I'm doing right
now too.


To clarify: you are NOT putting module-info.java in src/test/java.


Now it seems that I *must* add compile + test scoped to the *classpath*
to be able to compile the test classes.
My first approach was to leave the compile-scoped dependencies on the
modulepath and all test-scoped dependencies on the classpath, so the
modules keeps their inner related structure, but it seems that the
classpath classes cannot access the modulepath classes.


Your first approach sounds preferable. Can you copy-paste a minimized  
invocation of javac that works, and that doesn't work?


Alex


Here's the debug output when calling 'mvn test'.
This doesn't work:

[DEBUG] Classpath:
[DEBUG]   
E:\java-workspace\apache-maven-maven\maven\maven-builder-support\target\test-classes
[DEBUG]  C:\Users\Robert  
Scholte\.m2\repository\junit\junit\4.12\junit-4.12.jar
[DEBUG]  C:\Users\Robert  
Scholte\.m2\repository\org\hamcrest\hamcrest-core\1.3\hamcrest-core-1.3.jar

[DEBUG] Modulepath:
[DEBUG]   
E:\java-workspace\apache-maven-maven\maven\maven-builder-support\target\classes
[DEBUG]  C:\Users\Robert  
Scholte\.m2\repository\org\codehaus\plexus\plexus-utils\3.0.22\plexus-utils-3.0.22.jar
[DEBUG]  C:\Users\Robert  
Scholte\.m2\repository\org\apache\commons\commons-lang3\3.4\commons-lang3-3.4.jar

[DEBUG] Source roots:
[DEBUG]   
E:\java-workspace\apache-maven-maven\maven\maven-builder-support\src\test\java
[DEBUG]   
E:\java-workspace\apache-maven-maven\maven\maven-builder-support\target\generated-test-sources\test-annotations

[DEBUG] Command line options:
[DEBUG] -d  
E:\java-workspace\apache-maven-maven\maven\maven-builder-support\target\test-classes  
-classpath  
E:\java-workspace\apache-maven-maven\maven\maven-builder-support\target\test-classes;C:\Users\Robert  
Scholte\.m2\repository\junit\junit\4.12\junit-4.12.jar;C:\Users\Robert  
Scholte\.m2\repository\org\hamcrest\hamcrest-core\1.3\hamcrest-core-1.3.jar;  
-modulepath  
E:\java-workspace\apache-maven-maven\maven\maven-builder-support\target\classes;C:\Users\Robert  
Scholte\.m2\repository\org\codehaus\plexus\plexus-utils\3.0.22\plexus-utils-3.0.22.jar;C:\Users\Robert  
Scholte\.m2\repository\org\apache\commons\commons-lang3\3.4\commons-lang3-3.4.jar;  
-sourcepath  
E:\java-workspace\apache-maven-maven\maven\maven-builder-support\src\test\java;E:\java-workspace\apache-maven-maven\maven\maven-builder-support\target\generated-test-sources\test-annotations;  
-s  
E:\java-workspace\apache-maven-maven\maven\maven-builder-support\target\generated-test-sources\test-annotations  
-g -nowarn -target 1.9 -source 1.9 -encoding UTF-8


Result:
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]  
/E:/java-workspace/apache-maven-maven/maven/maven-builder-support/src/test/java/org/apache/maven/building/DefaultProblemTest.java:[25,41]  
package org.apache.maven.building.Problem does not exist
[ERROR]  
/E:/java-workspace/apache-maven-maven/maven/maven-builder-support/src/test/java/org/apache/maven/building/DefaultProblemCollectorTest.java:[24,41]  
package org.apache.maven.building.Problem does not exist


followed by a lot of 'cannot find symbol'.

If all modulepath entries were added to the classpath instead, then it  
compiles fine.


thanks,
Robert

Re: modulepath and classpath mixture

[Robert Scholte]

On Tue, 23 Feb 2016 13:59:13 +0100, Alan Bateman   
wrote:




On 22/02/2016 20:44, Robert Scholte wrote:

Hi,

first of all I'd like to say that I'm very pleased with the new -mp  
options, these matches better with the way Apache Maven would like to  
work with jars and class-folders.


Here's my use case: I noticed that if I add a module-info to  
src/main/java and put all compile-scoped dependencies to the module  
path, all compiles fines.
I assume that developers are less interested in adding a  
module-info.java file to src/test/java, so that's what I'm doing right  
now too.
Now it seems that I *must* add compile + test scoped to the *classpath*  
to be able to compile the test classes.
My first approach was to leave the compile-scoped dependencies on the  
modulepath and all test-scoped dependencies on the classpath, so the  
modules keeps their inner related structure, but it seems that the  
classpath classes cannot access the modulepath classes.


I'm looking for the confirmation that putting all dependencies on the  
classpath is indeed the right approach in this case.


For the tests then I assume they are in the same packages as the sources  
under src/main/java, is that right?


In that case I think you will want to compile the tests as if they are  
part of the module:


   javac  -Xmodule:m  -d testclasses/m  -mp m.jar  test/java/...

where m is the module name and the module (with sources in  
src/main/java) has already been compiled and then packaged as m.jar. The  
-Xmodule: option tells the compiler that you compiling the test  
classes as if they are part of module m. There is no module-info.java in  
the test tree.


The related lifecycle phases of Maven are: compile, test-compile, test,  
package.
So during test there's no m.jar yet, but target/classes or target/mods/m.  
This shouldn't be an issue, though.


If I understand this correctly I need to know the module name. That is  
information defined in the module-info, meaning I need to read that class  
first. When possible I would like to avoid this. Suppose a developer has  
made a syntax failure, I would hope that such error is thrown by javac,  
not by Maven while doing some pre-compile actions on the source-files to  
construct the correct commandline arguments.


I've already talked with Mark about the usage of -Xpatch, but that's  
required if src/test/java is considered a module too.


And maybe this is the key question: if src/main/java is a module, should  
we handle src/test/java as a module too or leave it as a classpath based  
project?


thanks,
Robert



Going further then I expect that JUnit or TestNG is also in the picture,  
I assume the class path. In that case, the command becomes:


   javac  -Xmodule:m  -d testclasses/m  -mp m.jar   \
   -cp junit-4.12.jar  -XaddReads:m=ALL-UNNAMED  \
   test/java/...

where you are compiling test classes as if they are module m and at the  
same time referencing JUnit types on the class path. The  
-XaddReads:m=ALL-UNNAMED augments the module declaration to say that  
module m reads all unnamed modules, just think class path here.



In order to run then you can use -Xpatch to augment the module with the  
test classes:


   java -Xpatch:testclasses  -mp m.jar  -cp junit-4.12.jar  
-XaddReads:m=ALL-UNNAMED   ...


It is as if the test classes are in m.jar. The alternative is of course  
to add the test classes to the packaged module but you would still need  
the -XaddReads because module m does not (and can not) declare that it  
depends on types on the class path.



While on the topic then I should mention that we have a proposal coming  
to support patches as JAR files as I'm sure you will get to the point  
soon where the test classes are in a JAR file.


Hopefully the above is useful. I completely agree with Jon that we need  
to put down detailed notes and examples. In the case of testing then we  
have tried out the popular test frameworks on the class path (as above)  
and also as modules. In the case of JUnit then we have been successful  
with it on a the module path as an automatic module. Definitely  
something to write up.


-Alan



--
Using Opera's mail client: http://www.opera.com/mail/

Re: modulepath and classpath mixture

[Jonathan Gibbons]

On 02/23/2016 12:48 PM, Robert Scholte wrote:
On Tue, 23 Feb 2016 01:52:50 +0100, Jonathan Gibbons 
 wrote:





On 02/22/2016 12:44 PM, Robert Scholte wrote:

Hi,

first of all I'd like to say that I'm very pleased with the new -mp 
options, these matches better with the way Apache Maven would like 
to work with jars and class-folders.


Here's my use case: I noticed that if I add a module-info to 
src/main/java and put all compile-scoped dependencies to the module 
path, all compiles fines.
I assume that developers are less interested in adding a 
module-info.java file to src/test/java, so that's what I'm doing 
right now too.
Now it seems that I *must* add compile + test scoped to the 
*classpath* to be able to compile the test classes.
My first approach was to leave the compile-scoped dependencies on 
the modulepath and all test-scoped dependencies on the classpath, so 
the modules keeps their inner related structure, but it seems that 
the classpath classes cannot access the modulepath classes.


I'm looking for the confirmation that putting all dependencies on 
the classpath is indeed the right approach in this case.


thanks,
Robert


Robert,

We definitely need some more detailed notes on setting up javac 
compilations (note to self!) but one thing to note is that by 
default, the unnamed module (i.e. code on the classpath) only has 
observability of the modules in the system image. To make modules on 
the module path observable, you need to use the -addmods option.


-- Jon


Hi Jonathan,

this would indeed explain what I'm facing right now. However, adding 
-addmods gives me the following exception:
Caused by: java.lang.IllegalArgumentException: -addmods requires an 
argument
at 
com.sun.tools.javac.main.Arguments.error(jdk.compiler@9-ea/Arguments.java:708)


Is -addmods followed by the same entries as -modulepath or by the 
modulenames. I really hope it is not the latter, because that would 
mean that I first need to discover and read all module-info files.


thanks,
Robert


Sorry, I should have been more explicit.

Both javac and java (the launcher) accept an option "-addmods 
,..." which can be used to name modules to be included in 
the module graph.   Confusingly, for javac, the option is listed under 
javac -X (that's a bug we will fix), but setting that aside, here's what 
the command line help says:


  -addmods [,...] Root modules to resolve in 
addition to the initial modules


"java -help" says effectively the same.


So yes, the option takes a list of module names, not module paths.

-- Jon

Re: modulepath and classpath mixture

[Robert Scholte]

On Tue, 23 Feb 2016 01:52:50 +0100, Jonathan Gibbons  
 wrote:





On 02/22/2016 12:44 PM, Robert Scholte wrote:

Hi,

first of all I'd like to say that I'm very pleased with the new -mp  
options, these matches better with the way Apache Maven would like to  
work with jars and class-folders.


Here's my use case: I noticed that if I add a module-info to  
src/main/java and put all compile-scoped dependencies to the module  
path, all compiles fines.
I assume that developers are less interested in adding a  
module-info.java file to src/test/java, so that's what I'm doing right  
now too.
Now it seems that I *must* add compile + test scoped to the *classpath*  
to be able to compile the test classes.
My first approach was to leave the compile-scoped dependencies on the  
modulepath and all test-scoped dependencies on the classpath, so the  
modules keeps their inner related structure, but it seems that the  
classpath classes cannot access the modulepath classes.


I'm looking for the confirmation that putting all dependencies on the  
classpath is indeed the right approach in this case.


thanks,
Robert


Robert,

We definitely need some more detailed notes on setting up javac  
compilations (note to self!) but one thing to note is that by default,  
the unnamed module (i.e. code on the classpath) only has observability  
of the modules in the system image. To make modules on the module path  
observable, you need to use the -addmods option.


-- Jon


Hi Jonathan,

this would indeed explain what I'm facing right now. However, adding  
-addmods gives me the following exception:
Caused by: java.lang.IllegalArgumentException: -addmods requires an  
argument
at  
com.sun.tools.javac.main.Arguments.error(jdk.compiler@9-ea/Arguments.java:708)


Is -addmods followed by the same entries as -modulepath or by the  
modulenames. I really hope it is not the latter, because that would mean  
that I first need to discover and read all module-info files.


thanks,
Robert

Re: modulepath and classpath mixture

[Jonathan Gibbons]

On 02/22/2016 12:44 PM, Robert Scholte wrote:

Hi,

first of all I'd like to say that I'm very pleased with the new -mp 
options, these matches better with the way Apache Maven would like to 
work with jars and class-folders.


Here's my use case: I noticed that if I add a module-info to 
src/main/java and put all compile-scoped dependencies to the module 
path, all compiles fines.
I assume that developers are less interested in adding a 
module-info.java file to src/test/java, so that's what I'm doing right 
now too.
Now it seems that I *must* add compile + test scoped to the 
*classpath* to be able to compile the test classes.
My first approach was to leave the compile-scoped dependencies on the 
modulepath and all test-scoped dependencies on the classpath, so the 
modules keeps their inner related structure, but it seems that the 
classpath classes cannot access the modulepath classes.


I'm looking for the confirmation that putting all dependencies on the 
classpath is indeed the right approach in this case.


thanks,
Robert


Robert,

We definitely need some more detailed notes on setting up javac 
compilations (note to self!) but one thing to note is that by default, 
the unnamed module (i.e. code on the classpath) only has observability 
of the modules in the system image. To make modules on the module path 
observable, you need to use the -addmods option.


-- Jon

Re: modulepath and classpath mixture

[Alex Buckley]

Hi Robert,

On 2/22/2016 12:44 PM, Robert Scholte wrote:

Here's my use case: I noticed that if I add a module-info to
src/main/java and put all compile-scoped dependencies to the module
path, all compiles fines.


Sounds good.


I assume that developers are less interested in adding a
module-info.java file to src/test/java, so that's what I'm doing right
now too.


To clarify: you are NOT putting module-info.java in src/test/java.


Now it seems that I *must* add compile + test scoped to the *classpath*
to be able to compile the test classes.
My first approach was to leave the compile-scoped dependencies on the
modulepath and all test-scoped dependencies on the classpath, so the
modules keeps their inner related structure, but it seems that the
classpath classes cannot access the modulepath classes.


Your first approach sounds preferable. Can you copy-paste a minimized 
invocation of javac that works, and that doesn't work?


Alex