Re: Avoiding same-package conflicts

[Jochen Theodorou]

I can second that by an example using Groovy. We have not yet taken 
steps to make jigsaw modules, but we quite some time ago we have split 
the code base in what would become eventually become modules in the 
future. Because of history this did mean to split packages. So we have a 
couple of jars (that want to be modules), with overlapping package names.


Examples of package conflicts with the "base module":

* The groovy-console has a conflict in groovy.ui
* groovy-ant has a conflict in groovy.util
* groovy-docgenerator in org.codehaus.groovy.tools
* groovy-nio in org.codehaus.groovy.runtime
* groovy-sql in org.codehaus.groovy.runtime
* groovy-swing in org.codehaus.groovy.runtime
* groovy-test in goovy.lang and groovy.util
* groovy-xml in org.codehaus.groovy.runtime and goovy.util

That's 7 "modules" with conflicts out of 19. One should remember here, 
that the codebase is like 10+ years old, and something that is now a 
"module" used to be fixed part of the language in the past. So some poor 
naming decision have been made. The org.codehaus.groovy.runtime 
conflicts could probably be solved, but the groovy.util ones are 
difficult, since this is a package imported by default (like java.lang 
in java), so moving classes to differing packages will break Groovy 
scripts... very elementary here for example GroovyTestCase in 
groovy-util, which is in the groovy-test "module"..


also... there is a automated meta class lookup system, that is based on 
the package name with a prefix. So someone could provide a meta class 
for java.util.ArrayList, while another does this for LinkedList. If they 
are using modules, they cannot be loaded at the same time. Granted, I 
don't like this mechanism, and I am looking for ways to deprecate it in 
the near future, but it is another example of same-package conflicts.


bye Jochen

On 30.10.2015 12:28, Stephen Colebourne wrote:

On 29 October 2015 at 13:48, Alan Bateman  wrote:

The restriction is that no two modules with the same package (irrespective
of whether it is exported or not, or who reads the modules) can be mapped to
the same class loader.


FWIW, I think that this could be a headache for module adoption.

Here are three cases that appear to be troubled by these restrictions:

- a large project that has taken an existing project (module) and
split it in two. In order to preserve backwards compatibility, the
author wants to retain the package names. But doing so is not
possible, because a single package now needs to be split across two
modules.

- extending a package scoped API from another project. While hardly
pleasant this is sometimes the only viable approach to get a task
done. As an example, the ImmutableList class from Google Guava has a
package scoped constructor. It is perfectly possible today for an
application to write a class that extends ImmutableList, so long as
that class is placed in the com.google.common.collect package. The
module system has the potential to block this extension point. (I
imagine that cglib and similar tools also generate code in other
peoples modules)

- where a stable interface is copied into two separate jar files to
avoid a dependency. An example of this was commons-beanutils where 4
interfaces were copied from commons-collections. While this has now
been changed, it was considered useful at the time, and given the
stability of the interfaces in question, caused no problems.
https://issues.apache.org/jira/browse/BEANUTILS-379

FWIW, I'm most concerned about the first one going forward. However,
the other two are cases I've seen used, and if used broadly in the
wild could slow adoption (if it is hard to make those jar files
modular, and they are widely depended on).

Stephen

Re: Avoiding same-package conflicts

[Stephen Colebourne]

On 29 October 2015 at 13:48, Alan Bateman  wrote:
> The restriction is that no two modules with the same package (irrespective
> of whether it is exported or not, or who reads the modules) can be mapped to
> the same class loader.

FWIW, I think that this could be a headache for module adoption.

Here are three cases that appear to be troubled by these restrictions:

- a large project that has taken an existing project (module) and
split it in two. In order to preserve backwards compatibility, the
author wants to retain the package names. But doing so is not
possible, because a single package now needs to be split across two
modules.

- extending a package scoped API from another project. While hardly
pleasant this is sometimes the only viable approach to get a task
done. As an example, the ImmutableList class from Google Guava has a
package scoped constructor. It is perfectly possible today for an
application to write a class that extends ImmutableList, so long as
that class is placed in the com.google.common.collect package. The
module system has the potential to block this extension point. (I
imagine that cglib and similar tools also generate code in other
peoples modules)

- where a stable interface is copied into two separate jar files to
avoid a dependency. An example of this was commons-beanutils where 4
interfaces were copied from commons-collections. While this has now
been changed, it was considered useful at the time, and given the
stability of the interfaces in question, caused no problems.
https://issues.apache.org/jira/browse/BEANUTILS-379

FWIW, I'm most concerned about the first one going forward. However,
the other two are cases I've seen used, and if used broadly in the
wild could slow adoption (if it is hard to make those jar files
modular, and they are widely depended on).

Stephen

Re: Avoiding same-package conflicts

[David M. Lloyd]

On 10/30/2015 04:12 AM, Martin Lehmann wrote:

Hi David, hi all,

thanks, David, for your response.


Sure, reply is inline.


Full ACK. Bad practice.

I disagree, actually.  I think that this is a completely needless and 
artificial restriction that arose from implementation decisions, not from a 
valid requirement.

You have a point. I don't disagree ;-)


So we really need disjunct classes in *all* libraries now? Not even, if the redundant 
packages are not even exported (right?). Would it work in the "old" classpath?

I think that judging by whether something would work in an old classpath is a 
false test.


I see your point, but:
On the one hand: Migration will be a big issue. I followed the Monday JavaOne 
live streams talks (@Alan, @Alex: Thanks for the great talks!) on the topic and 
understood that stuff should work like before.
Well, ambitious enough, but if this is indeed the goal, then "did it work in the old 
classpath?" is definitely something to look at.
On the other hand - don't get me wrong: I personally prefer to have a clean and 
 proper solution without (too much) of bad design *just* because backward 
compatibility.


Sure, but only because you've reversed the logical sense of my argument: 
 I'm saying, don't limit what works under modules to what worked under 
classpath (i.e. using "well it never worked before" as a first-order 
justification for not allowing something to work under modules, or not 
caring that something doesn't work under modules).  You're saying, don't 
break things that previously worked on classpath.  In other words, we're 
not disagreeing, we're making two different points, and I agree with 
yours as well.



The whole point of modules is to throw off the restrictions and problems of the 
classpath, after all.  I think a better test is, "does this make sense?", and 
to use your slf4j example - it definitely makes sense, because:
...

Yep, agree.

Btw, I meanwhile thought of a second use case which will *definitely* be 
needed. Assume to use a third-party library where something is wrong. Fix is 
not (yet) available, so you need to fix something locally. Up to now a common 
though bad(?) practice is to patch the library class(es) locally and put it the 
patched class(es) on the classpath *before* the third-party library (... as a 
workaround until the real fix is there).

I doubt that I am the only one who ever had the need for such a workaround. If 
redundant packages are not longer allowed, then I (we?) need a replacement for 
such a scenario. (Or are I am now really expected to either encapsulate 
external stuff in different layers and/or to repackage external jar files to 
guarantee disjunct contents...?) Makes sense?


Under the JBoss Modules system, you typically drop your fixed JAR 
alongside the broken one in the module path, then update your module.xml 
file to point at the new one instead of the old one.  But this only 
works because module descriptors are external to the module in JBoss 
Modules.


--
- DML

RE: Avoiding same-package conflicts

[Martin Lehmann]

Hi David, hi all,

thanks, David, for your response.

>> Full ACK. Bad practice.
>I disagree, actually.  I think that this is a completely needless and 
>artificial restriction that arose from implementation decisions, not from a 
>valid requirement.
You have a point. I don't disagree ;-)

>> So we really need disjunct classes in *all* libraries now? Not even, if the 
>> redundant packages are not even exported (right?). Would it work in the 
>> "old" classpath?
> I think that judging by whether something would work in an old classpath is a 
> false test.  

I see your point, but:
On the one hand: Migration will be a big issue. I followed the Monday JavaOne 
live streams talks (@Alan, @Alex: Thanks for the great talks!) on the topic and 
understood that stuff should work like before. 
Well, ambitious enough, but if this is indeed the goal, then "did it work in 
the old classpath?" is definitely something to look at. 
On the other hand - don't get me wrong: I personally prefer to have a clean and 
 proper solution without (too much) of bad design *just* because backward 
compatibility.

> The whole point of modules is to throw off the restrictions and problems of 
> the classpath, after all.  I think a better test is, "does this make sense?", 
> and to use your slf4j example - it definitely makes sense, because:
> ...
Yep, agree.

Btw, I meanwhile thought of a second use case which will *definitely* be 
needed. Assume to use a third-party library where something is wrong. Fix is 
not (yet) available, so you need to fix something locally. Up to now a common 
though bad(?) practice is to patch the library class(es) locally and put it the 
patched class(es) on the classpath *before* the third-party library (... as a 
workaround until the real fix is there). 

I doubt that I am the only one who ever had the need for such a workaround. If 
redundant packages are not longer allowed, then I (we?) need a replacement for 
such a scenario. (Or are I am now really expected to either encapsulate 
external stuff in different layers and/or to repackage external jar files to 
guarantee disjunct contents...?) Makes sense?

Cheers,
Martin

-Original Message-
From: jigsaw-dev [mailto:jigsaw-dev-boun...@openjdk.java.net] On Behalf Of 
David M. Lloyd
Sent: Thursday, October 29, 2015 12:33 PM
To: jigsaw-dev@openjdk.java.net
Subject: Re: Avoiding same-package conflicts

On 10/29/2015 05:59 AM, Martin Lehmann wrote:
> Hi all,
>
> I stumbled across the same "issue".
>
>> that having multiple versions of the same library isn't a best practice to 
>> say the least.
> Full ACK. Bad practice.

I disagree, actually.  I think that this is a completely needless and 
artificial restriction that arose from implementation decisions, not 
from a valid requirement.

We have used our ability to ship multiple modules with the same package 
names to good effect on more than one occasion.  As long as a given 
module doesn't link the conflicting packages, there is no real problem. 
  Even hypothetical issues involving indirect linkage or passing objects 
back and forth have never really been a problem for us.

> I might have an (artifical?) use case where this might lead to some extra 
> effort in migration, though:
> Let's assume that two libraries D1 and D2 implement the same common API. For 
> some historical reason, both libraries *ship* the interfaces classes of that 
> common API.
> This won't work (easily) without repackaging jar files (even if the common 
> API classes would not even differ).
>
> Example:
> A --requires --> B, C
> B --requires --> D1
> C --requires --> D2
> D1 and D2 both contain the same interfaces. Even if neither B nor C would 
> *not* "requires public" D1 / D2, this would not work.
>
> Too artifical? I actually thought of two logging implementations using & 
> shipping the common slf4j interface classes.
>
> So we really need disjunct classes in *all* libraries now? Not even, if the 
> redundant packages are not even exported (right?). Would it work in the "old" 
> classpath?

I think that judging by whether something would work in an old classpath 
is a false test.  The whole point of modules is to throw off the 
restrictions and problems of the classpath, after all.  I think a better 
test is, "does this make sense?", and to use your slf4j example - it 
definitely makes sense, because:

* Modules generally use slf4j as a private API to perform their own 
logging functions
* Thus slf4j generally never leaks to a module's exported package set
* Thus there's no actual harm in allowing this, and much benefit - 
especially if B and C (in your example) are not completely under your 
control (which is inevitable in any nontrivial system)

-- 
- DML

JOSM feedback on Java 7,8,9, including Jigsaw EA

[Vincent Privat]

Hi,
Following the recent inclusion of JOSM to the Quality Outreach list by the
Adoption Group, we have compiled every single Java issue we have
encountered, and reported when it was new, for the latest versions of Java,
on a single page:

https://josm.openstreetmap.de/wiki/JavaBugs

We will use this page to coordinate our communication efforts with you. Is
it possible to add a link to it in the last column of the Quality Outreach
table?

The list currently contains 29 unresolved items and 18 resolved ones.

I won't go through all of them in this e-mail. Ideally we'd like to see all
of them fixed in a future Java release but I will only focus on the most
important ones.

Concerning Jigsaw:
- We have reported 3 bugs. All made it to the public JIRA: 8138878,
8140477, 8140481. The second one is a bit problematic for our tests as it
basically kills our Jenkins instance. I see the two other ones are
understood/in progress. We will do more tests after we resolve the hanging
problem.
- We'd like to know if it can be expected to see the
package sun.security.x509 become a public JDK API, for example in
javax.security.cert? We currently use it to generate a self-signed
certificate in order to create a local https server. That's our only use of
private JDK API.

Concerning Valhalla:
- I see some discussions about building the JDK with project Valhalla. Are
you going to provide public Early Access builds like project Jigsaw?

Concerning the JIRA database:
- Is it possible to add the label "josm-found" to
issues 8140481, 8139659, 8034224, 7158257, 7194099 ?
- Some issues didn't make it to the public JIRA and remained in the private
bug database. Can we please have more information on them (why have they
apparently been rejected)? The incident numbers
are JI-9009025, JI-9010791, JI-9009449, JI-9008003.
- Is it expected to allow external people to have the possibility to
subscribe to JDK issues?

Concerning our incoming migration from Java 7 to Java 8:
- I am concerned about three issues in Java2D/AWT on Linux. We have several
duplicate bug reports for them: 6322854, 7172749, 8098530. Can we hope for
a fix in a future update of Java 8?

Finally:
- We had a terrible experience when trying to report a bug against JAXP. We
detected a severe data corruption problem in StaX when dealing with Unicode
SMP characters, so we reported it, including a sample Java program 100%
reproducible, in January 2013 (JAXP-76 on java.net JIRA). As no activity
was visible on this JIRA instance, we tried to use the standard Java bug
report, three times, without success, with incident numbers 2431783
(2013-01-23), 2627098 (2013-10-28) and 9048481 (2014-11-28), without any
answer. On 2014, November 29th we discovered by chance that the bug had
finally been detected and fixed internally, as JDK-8058175 (created and
resolved in September 2014). We reported back to the public JAXP JIRA
instance, again without any answer. 6 months later we finally got the
ironic and laconic answer "Please report issues to the OpenJDK Bug System",
which was exactly was we were trying to do for 2 years! Can you please tell
us why our bug reports were all silently ignored while the bug was real,
and if is it still worth reporting bugs against JAXP? Thankfully we had far
better experiences with other components of the JDK.

Re: JOSM feedback on Java 7,8,9, including Jigsaw EA

[Rory O'Donnell]

On 30/10/2015 15:29, Vincent Privat wrote:

Hi,
Following the recent inclusion of JOSM to the Quality Outreach list by 
the Adoption Group, we have compiled every single Java issue we have 
encountered, and reported when it was new, for the latest versions of 
Java, on a single page:


https://josm.openstreetmap.de/wiki/JavaBugs

We will use this page to coordinate our communication efforts with 
you. Is it possible to add a link to it in the last column of the 
Quality Outreach table?
Added to the wiki, thanks for logging all these issues. I suggest you 
followup with the
relevant mailing lists to get specific feedback as you have done with 
jigsaw-dev.




The list currently contains 29 unresolved items and 18 resolved ones.

I won't go through all of them in this e-mail. Ideally we'd like to 
see all of them fixed in a future Java release but I will only focus 
on the most important ones.


Concerning Jigsaw:
- We have reported 3 bugs. All made it to the public JIRA: 8138878, 
8140477, 8140481. The second one is a bit problematic for our tests as 
it basically kills our Jenkins instance. I see the two other ones are 
understood/in progress. We will do more tests after we resolve the 
hanging problem.
- We'd like to know if it can be expected to see the 
package sun.security.x509 become a public JDK API, for example in 
javax.security.cert? We currently use it to generate a self-signed 
certificate in order to create a local https server. That's our only 
use of private JDK API.


Concerning Valhalla:
- I see some discussions about building the JDK with project Valhalla. 
Are you going to provide public Early Access builds like project Jigsaw?


Concerning the JIRA database:
- Is it possible to add the label "josm-found" to 
issues 8140481, 8139659, 8034224, 7158257, 7194099 ?
- Some issues didn't make it to the public JIRA and remained in the 
private bug database. Can we please have more information on them (why 
have they apparently been rejected)? The incident numbers 
are JI-9009025, JI-9010791, JI-9009449, JI-9008003.
- Is it expected to allow external people to have the possibility to 
subscribe to JDK issues?

I will take a look at these and come back to you.


Concerning our incoming migration from Java 7 to Java 8:
- I am concerned about three issues in Java2D/AWT on Linux. We have 
several duplicate bug reports for them: 6322854, 7172749, 8098530. Can 
we hope for a fix in a future update of Java 8?


Finally:
- We had a terrible experience when trying to report a bug against 
JAXP. We detected a severe data corruption problem in StaX when 
dealing with Unicode SMP characters, so we reported it, including a 
sample Java program 100% reproducible, in January 2013 (JAXP-76 on 
java.net  JIRA). As no activity was visible on this 
JIRA instance, we tried to use the standard Java bug report, three 
times, without success, with incident numbers 2431783 
(2013-01-23), 2627098 (2013-10-28) and 9048481 (2014-11-28), without 
any answer. On 2014, November 29th we discovered by chance that the 
bug had finally been detected and fixed internally, as JDK-8058175 
(created and resolved in September 2014). We reported back to the 
public JAXP JIRA instance, again without any answer. 6 months later we 
finally got the ironic and laconic answer "Please report issues to the 
OpenJDK Bug System", which was exactly was we were trying to do for 2 
years! Can you please tell us why our bug reports were all silently 
ignored while the bug was real, and if is it still worth reporting 
bugs against JAXP? Thankfully we had far better experiences with other 
components of the JDK.


I will take a look for you and see what I can find.

Rgds,Rory

--
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland

Re: JOSM feedback on Java 7,8,9, including Jigsaw EA

[Mandy Chung]

Thanks for reporting these issues.

> On Oct 30, 2015, at 8:29 AM, Vincent Privat  
> wrote:
> 
> Concerning Jigsaw:
> - We have reported 3 bugs. All made it to the public JIRA: 8138878,
> 8140477, 8140481. The second one is a bit problematic for our tests as it
> basically kills our Jenkins instance. I see the two other ones are
> understood/in progress. We will do more tests after we resolve the hanging
> problem.

8140481 is a build issue that should be fixed shortly.

8138878 breaks through resource encapsulation.  jake b86 fixes InternalError 
and throws MissingResourceException if it attempts to get a bundle in 
java.desktop named module.

We’ll look into these bugs.

> 
> - We'd like to know if it can be expected to see the
> package sun.security.x509 become a public JDK API, for example in
> javax.security.cert? We currently use it to generate a self-signed
> certificate in order to create a local https server. That's our only use of
> private JDK API.

There are two RFEs related to signing and certificates
  8058778: New APIs for some keytool functions
  8056174: New APIs for jar signing

I have added this comment in 8058778 for the security team to look into.  You 
can subscribe to security-...@openjdk.java.net  where the discussion for these 
RFEs will be.

Mandy

Re: JOSM feedback on Java 7,8,9, including Jigsaw EA

[Alan Bateman]

On 30/10/2015 15:29, Vincent Privat wrote:

:

Concerning Jigsaw:
- We have reported 3 bugs. All made it to the public JIRA: 8138878, 
8140477, 8140481. The second one is a bit problematic for our tests as 
it basically kills our Jenkins instance. I see the two other ones are 
understood/in progress. We will do more tests after we resolve the 
hanging problem.
JDK-8138878 seems to be JSOM hacking into private field in 
java.awt.Toolkit. It is possible to discuss the issue on awt-dev and 
i18n-dev to see if the changing locale scenario is meant to work?


We'll look into JDK-8140477. Note that there has been deadlocks 
involving instrumentation going back to JDK 5 (when java.lang.instrument 
was introduced). I don't know if this one is in that bucket but 
generally agents need to be take great care when instrumenting code 
defined to the ext and boot loader (because the agent's classes are 
defined by the app class loader).


JDK-8140481 seem to be the jdk.management module missing from the JDK 9 
build, it may not be specific to the Jigsaw EA builds.





:

Concerning Valhalla:
- I see some discussions about building the JDK with project Valhalla. 
Are you going to provide public Early Access builds like project Jigsaw?



This is something to ask on valhalla-dev, it might be too early for that.

-Alan

Re: JOSM feedback on Java 7,8,9, including Jigsaw EA

[Jaroslav Bachorik]

On 30.10.2015 18:02, Alan Bateman wrote:



On 30/10/2015 15:29, Vincent Privat wrote:
> :
>
> Concerning Jigsaw:
> - We have reported 3 bugs. All made it to the public JIRA: 8138878,
> 8140477, 8140481. The second one is a bit problematic for our tests as
> it basically kills our Jenkins instance. I see the two other ones are
> understood/in progress. We will do more tests after we resolve the
> hanging problem.
JDK-8138878 seems to be JSOM hacking into private field in
java.awt.Toolkit. It is possible to discuss the issue on awt-dev and
i18n-dev to see if the changing locale scenario is meant to work?

We'll look into JDK-8140477. Note that there has been deadlocks
involving instrumentation going back to JDK 5 (when java.lang.instrument
was introduced). I don't know if this one is in that bucket but
generally agents need to be take great care when instrumenting code
defined to the ext and boot loader (because the agent's classes are
defined by the app class loader).

JDK-8140481 seem to be the jdk.management module missing from the JDK 9
build, it may not be specific to the Jigsaw EA builds.


No, it's not. Same problem with non-jigsaw builds.

-JB-





> :
>
> Concerning Valhalla:
> - I see some discussions about building the JDK with project Valhalla.
> Are you going to provide public Early Access builds like project Jigsaw?
>
This is something to ask on valhalla-dev, it might be too early for that.

-Alan

Re: Avoiding same-package conflicts

[Alan Bateman]

On 30/10/2015 11:28, Stephen Colebourne wrote:

:

Here are three cases that appear to be troubled by these restrictions:

- a large project that has taken an existing project (module) and
split it in two. In order to preserve backwards compatibility, the
author wants to retain the package names. But doing so is not
possible, because a single package now needs to be split across two
modules.
The module system can facilitate many refactoring scenarios. This one 
seems an incompatible change. Even if the module system supported 
splitting a package between modules then it would force consumers of the 
original module to change their module dependences.




- extending a package scoped API from another project. While hardly
pleasant this is sometimes the only viable approach to get a task
done. As an example, the ImmutableList class from Google Guava has a
package scoped constructor. It is perfectly possible today for an
application to write a class that extends ImmutableList, so long as
that class is placed in the com.google.common.collect package. The
module system has the potential to block this extension point. (I
imagine that cglib and similar tools also generate code in other
peoples modules)
I assume if Guava wanted ImmutableList to be part of its API then it 
would have made it public. In any case, testing a package private class 
is a scenario that I can relate to. The -Xpatch option can help with 
that. In the intro slide deck then there is an example, it's the in 
"Director's cut", meaning the bonus slides after the presentation end slide.





- where a stable interface is copied into two separate jar files to
avoid a dependency. An example of this was commons-beanutils where 4
interfaces were copied from commons-collections. While this has now
been changed, it was considered useful at the time, and given the
stability of the interfaces in question, caused no problems.
https://issues.apache.org/jira/browse/BEANUTILS-379

Going forward then the ability to express dependences clearly should 
help avoid the need to do this kind of thing.


-Alan.

Re: JOSM feedback on Java 7,8,9, including Jigsaw EA

[Vincent Privat]

2015-10-30 18:02 GMT+01:00 Alan Bateman :

>
> JDK-8138878 seems to be JSOM hacking into private field in
> java.awt.Toolkit. It is possible to discuss the issue on awt-dev and
> i18n-dev to see if the changing locale scenario is meant to work?
>
>
Yes, this is a hack we found in the Sun bug database as a workaround
to JDK-6292739.
The bug describes the changing locale scenario:

"If you change the default locale at runtime, the text displayed for
accelerator keys doesn't change, e.g. if you start with default locale
Locale.GERMAN an accelerator for "control I", which shows up as "Strg-I",
doesn't change to "Ctrl-I" when changing the Locale to Locale.US."

As JDK-6292739 is classified to "Bug" we assume it is meant to work. If
this bug is resolved we can remove the hack from JOSM.