Hi OpenJDK Core Developers,
you may know the Apache Lucene team is testing early access releases of
Java 9. We reported many bugs already, but most of them only applied to
Hotspot and Lucene itsself. But this problem since build 108 is now
really severe, because it breaks the build system already!
To allow further testing of Open Source Projects, I'd suggest to revert
the Multi-Release-JAR runtime support patch and provide a new preview
build ASAP, because we found out after a night of debugging a build
system from which we don't know all internals what is causing the
problems and there is no workaround. I am very sorry that I have to say
this, but it unfortunately build 108 breaks *ALL* versions of Apache
Ant, the grandfather of all Java build systems :-) I know also OpenJDK
is using it, too! So with Multi-Release JAR file patch applied (see
http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/f9913ea0f95c), any
Ant-based build - including the JDK build itsself - would no longer
bootstrap. It is impossible to also build Gradle projects, because
Gradle uses Ant internally for many tasks). Maven projects may be
affected, too.
Now you might have the question: What happened?
We tried to build Lucene on our Jenkins server, but the build itsself
failed with a stupid error message:
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:21: The
following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:56:
not doesn't support the nested "matches" element.
The first idea was: Ah, there were changes in XML parsing
(JDK-8149915). So we debugged the build. But it was quite clear that
XML parsing was not the issue. It got quite clear when we enabled
"-debug" on the build. What happened was that Ant was not loading its
internal conditions/tasks/type definitions anymore, so the build system
does not know almost any type anymore. The debug log showed that Ant
was no longer able to load the resource
"/org/apache/tools/ant/antlib.xml" from its own JAR file anymore.
Instead it printed some strange debugging output (which looked totally
broken).
I spend the whole night digging through their code and found the issue:
The commit of Multi-Release-Jar files (see
http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/f9913ea0f95c) broke
resource handling in Apache Ant. In short: If you call
ClassLoader.getResources() / or getResource() you get back an URL from
where you can load the Resource - this is all fine and still works.
But, with the Multi-Release JAR files patch this now has an URL
fragment appended to the URL: '#release' (see
http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/f9913ea0f95c); this also
applies to non-multi-release JAR files like Apache Ant's "ant.jar".
In Java 7, Java 8,... and Java 9pre-b108,
ClassLoader.getResource()/getResources() returned stuff like:
"jar:file:/C:/Program%20Files/Java/apache-ant-1.9.6/lib/ant.jar!/org/apache/tools/ant/antlib.xml"
Now in Java 9b108 the following is returned:
"jar:file:/C:/Program%20Files/Java/apache-ant-1.9.6/lib/ant.jar!/org/apache/tools/ant/antlib.xml#release"
And here Ant breaks (and I assume many other projects like Maven, too).
Ant checks for the file extension of the string (because it may load
definitions from both XML and properties files). So it does
endsWith(".xml") and of course this now returns false. The effect is
that Ant tries to load its own task definitions as a java properties
file instead of XML. Of course this fails, because the data behind this
URL is XML. The effect is that Ant cannot bootstrap as everything to
build is missing.
One might say: Ant's code is broken (I agree, it is not nice because it
relies on the string representation of the resource URL - which is a
no-go anyways), but it is impossible to fix, because Ant is bundled on
most developer computers and those will suddenly break with Java 9!
There is also no version out there that works around this, so we cannot
test anything anymore!
The problematic line in Ant's code is here:
http://grepcode.com/file/repo1.maven.org/maven2/org.apache.ant/ant/1.9.6/org/apache/tools/ant/taskdefs/Definer.java?av=f#259
I'd suggest to please ASAP revert the Multi-Release JAR file patch and
provide a new preview build as soon as possible. I think there is more
work needed to fix this. If this does not revert to the original state,
it will be impossible to build and test Lucene, Elasticsearch,.... (and
almost every Java project out there!). So short: We cannot test anymore
and it is likely that we cannot support Java 9 anymore because the
build system used by most Java projects behind the scenes does not
bootstrap itself anymore.
My suggestion would be to investigate other versions for this patch
that does *not* modify the resource URLs by appending a fragment to
them (at least not for the "standard" case without an actual
Multi-Release Jar). For new multi-release JAR files I am fine with
appending fragments, but please not for default ones. Maybe change code
to handle the URLs from the non-versioned part differently (without
fragment). Leaving the fragment inide may break many othe rprojects,
because many programmers are very sloppy with handling URLs (well-known
issue is calling URL#getFile() of a file:-URL that breaks on Windows
systems and spaces in path name). Many people just call toString() on
URL and do some test on it (startsWith, endsWith). So appending
fragments is a no-go for backwards compatibility with JAR resources!
I posted this to the mailing list and did not open a bug report on
http://bugs.java.com/, because this is a more general issue - feel free
to open bug reports around this!!! I would be very happy if we could
find a quick solution for this problem. Until there is a solution we
have to stop testing Java 9 with Apache Lucene/Solr/..., and this is
not a good sign, especially as Jigsaw will be merged soon.
Thanks for listening,
Uwe
P.S.: I also CCed the Apache Ant team. They should fix the broken code
anyways, but this won't help for many projects already out there (e.g.
Apache Lucene still has a minimum requirement of Ant 1.8.2 because
MacOSX computers ship with that version since years).
-----
Uwe Schindler
uschind...@apache.org
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/
