Hi
This (long) email reports a defect, plus the source code explanation for
the defect,
plus a proposed source code change (which I have tested) to correct the
defect.
This email is formatted in a fixed-width font and should be read in a wide
window
(about 100 characters except for some longer lines).
FAILURE REPORT
An attempt to use the Ivy install task to install a module from a Maven
repository where the module POM has a DOCTYPE declaration fails.
The following request is an example demonstrating the failure:
<ivy:install organisation="com.vaadin.addon"
module="vaadin-charts"
revision="3.3.4"
from="vaadin-addons"
to="third-party-local"
transitive="false"
overwrite="true"/>
This produces the following failure:
[ivy:install] :: installing com.vaadin.addon#vaadin-charts;3.3.4 ::
[ivy:install] :: resolving dependencies ::
[ivy:install] [Fatal Error] ivy-3.3.4.xml.original:4:10: Already seen
doctype.
[ivy:install] :: downloading artifacts to cache ::
[ivy:install] :: installing in third-party-local ::
[ivy:install] :: install resolution report ::
[ivy:install] :: resolution report :: resolve 0ms :: artifacts dl 0ms
---------------------------------------------------------------------
| | modules ||
artifacts |
| conf | number| search|dwnlded|evicted||
number|dwnlded|
---------------------------------------------------------------------
| default | 1 | 0 | 0 | 0 || 0 |
0 |
---------------------------------------------------------------------
[ivy:install]
[ivy:install] :: problems summary ::
[ivy:install] :::: WARNINGS
[ivy:install] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:install] :: UNRESOLVED DEPENDENCIES ::
[ivy:install] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:install] :: com.vaadin.addon#vaadin-charts;3.3.4:
java.text.ParseException: Already
seen doctype.
[ivy:install] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:install] :::: ERRORS
[ivy:install] Already seen doctype.
[ivy:install]
[ivy:install] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
BUILD FAILED
The "vaadin-addons" resolver is defined as:
<ibiblio name="vaadin-addons" usepoms="true" useMavenMetadata="true"
m2compatible="true" root="
https://maven.vaadin.com/vaadin-addons"/>
DEFECT DIAGNOSIS
Running with "-debug" shows following as root cause exception:
[ivy:install] Caused by: org.xml.sax.SAXParseException; systemId:
file:/home/colin.chambers/.ivy2/cache/com.vaadin.addon/vaadin-charts/ivy-3.3.4.xml.original;
lineNumber: 4; columnNumber: 10; Already seen doctype.
[ivy:install] at
java.xml/com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:262)
[ivy:install] at
java.xml/com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:342)
[ivy:install] at
org.apache.ivy.util.XMLHelper.parseToDom(XMLHelper.java:256)
[ivy:install] at
org.apache.ivy.plugins.parser.m2.PomReader.<init>(PomReader.java:129)
[ivy:install] at
org.apache.ivy.plugins.parser.m2.PomModuleDescriptorParser.parseDescriptor(PomModuleDescriptorParser.java:140)
The file "ivy-3.3.4.xml.original" referenced in the cache starts with:
1: <?xml version="1.0" encoding="UTF-8"?>
2: <!DOCTYPE project>
3: <project xmlns="https://maven.apache.org/POM/4.0.0"; xmlns:xsi="
https://www.w3.org/2001/XMLSchema-instance";
4: xsi:schemaLocation="https://maven.apache.org/POM/4.0.0
https://maven.apache.org/xsd/maven-4.0.0.xsd";>
5:
6: <parent>
7: <groupId>com.vaadin.addon</groupId>
8: <artifactId>vaadin-charts-parent</artifactId>
9: <version>3.3.4</version>
10: </parent>
Note that line 4 is *not* a DOCTYPE declaration.
This file may also be observed in the "vaadin-addons" repository at URL
https://maven.vaadin.com/vaadin-addons/com/vaadin/addon/vaadin-charts/3.3.4/vaadin-charts-3.3.4.pom
(Use "view source" to see the prolog and DOCTYPE declaration.)
The source code around line number 129 of PomReader (from stack trace
above) is as follows:
public PomReader(final URL descriptorURL, final Resource res) throws
IOException, SAXException {
InputStream stream = new AddDTDFilterInputStream(
URLHandlerRegistry.getDefault().openStream(descriptorURL));
InputSource source = new InputSource(stream);
source.setSystemId(XMLHelper.toSystemId(descriptorURL));
try {
129: Document pomDomDoc = XMLHelper.parseToDom(source, new
EntityResolver() {
public InputSource resolveEntity(String publicId, String
systemId)
throws SAXException, IOException {
if (systemId != null &&
systemId.endsWith("m2-entities.ent")) {
// IVY-921: return an InputSource for our local
packaged m2-entities.ent file
return new InputSource(
PomReader.class.getResourceAsStream("m2-entities.ent"));
}
return null;
}
}, true, XMLHelper.ExternalResources.IGNORE);
Line 129 invokes an XML parser on the source. Just above this the
InputStream can be seen
to be filtered by class AddDTDFilterInputStream. This class is defined in
PomReader and
the code up to and including the constructor definition is as follows (line
797 on):
private static final class AddDTDFilterInputStream extends
FilterInputStream {
private static final int MARK = 10000;
private static final String DOCTYPE = "<!DOCTYPE project SYSTEM
\"m2-entities.ent\">\n";
private int count;
private byte[] prefix = DOCTYPE.getBytes();
private AddDTDFilterInputStream(InputStream in) throws IOException {
super(new BufferedInputStream(in));
this.in.mark(MARK);
// TODO: we should really find a better solution for this...
// maybe we could use a FilterReader instead of a
FilterInputStream?
int byte1 = this.in.read();
int byte2 = this.in.read();
int byte3 = this.in.read();
if (byte1 == 239 && byte2 == 187 && byte3 == 191) {
// skip the UTF-8 BOM
this.in.mark(MARK);
} else {
this.in.reset();
}
824: int bytesToSkip = 0;
LineNumberReader reader = new LineNumberReader(new
InputStreamReader(this.in, StandardCharsets.UTF_8),
100);
String firstLine = reader.readLine();
if (firstLine != null) {
String trimmed = firstLine.trim();
if (trimmed.startsWith("<?xml ")) {
int endIndex = trimmed.indexOf("?>");
String xmlDecl = trimmed.substring(0, endIndex + 2);
prefix = (xmlDecl + "\n" + DOCTYPE).getBytes();
bytesToSkip = xmlDecl.getBytes().length;
}
} else {
prefix = new byte[0];
}
this.in.reset();
for (int i = 0; i < bytesToSkip; i++) {
this.in.read();
}
}
...
>From line 824 ("int bytesToSkip = 0") onwards, if the document starts with
an "<?xml"
prolog then a prefix is constructed containg the prolog and a DOCTYPE
declaration
(defined on the first line above). The DOCTYPE declaration is inserted
whether or
not the document already contains a DOCTYPE declaration. The code below
(not shown)
inserts the prefix in the stream returned by the "read()" methods.
Therefore if the document already contained a DOCTYPE declaration (as in
the example)
then the document returned by this filter class contains two DOCTYPE
declarations,
thus causing the "Already seen doctype" ParseException.
The following is the start of the output from this code for the example POM
above:
1: <?xml version="1.0" encoding="UTF-8"?>
2: <!DOCTYPE project SYSTEM "m2-entities.ent">
3:
4: <!DOCTYPE project>
5: <project xmlns="https://maven.apache.org/POM/4.0.0"; xmlns:xsi="
https://www.w3.org/2001/XMLSchema-instance";
6: xsi:schemaLocation="https://maven.apache.org/POM/4.0.0
https://maven.apache.org/xsd/maven-4.0.0.xsd";>
7:
8: <parent>
9: <groupId>com.vaadin.addon</groupId>
10: <artifactId>vaadin-charts-parent</artifactId>
11: <version>3.3.4</version>
12: </parent>
Note the blank line 3 (caused by inaccurate calculation of bytes_to_skip)
and
the second DOCTYPE on line 4 (as reported in the failure).
PROPOSED SOLUTION
Replace above code with the following:
private static final class AddDTDFilterInputStream extends
FilterInputStream {
private static final int MARK = 10000;
private static final String DOCTYPE = "<!DOCTYPE project SYSTEM
\"m2-entities.ent\">\n";
private int count;
private byte[] prefix;
private AddDTDFilterInputStream(InputStream in) throws IOException {
super(new BufferedInputStream(in));
this.in.mark(MARK);
// TODO: we should really find a better solution for this...
// maybe we could use a FilterReader instead of a
FilterInputStream?
int byte1 = this.in.read();
int byte2 = this.in.read();
int byte3 = this.in.read();
if (byte1 == 239 && byte2 == 187 && byte3 == 191) {
// skip the UTF-8 BOM
this.in.mark(MARK);
} else {
this.in.reset();
}
// Read prefix of document up to and including any DOCTYPE
declaration.
// Construct replacement prefix by inserting POM DOCTYPE and
removing
// existing DOCTYPE, if present.
StringBuilder prefixBuilder = new StringBuilder();
int lines_read = 0;
LineNumberReader reader = new LineNumberReader(new
InputStreamReader(this.in, StandardCharsets.UTF_8),
100);
String line = reader.readLine();
++lines_read;
if (line != null) {
line = line.trim();
if (line.startsWith("<?xml ")) {
prefixBuilder.append(line + "\n");
line = reader.readLine();
++lines_read;
}
prefixBuilder.append(DOCTYPE);
if (line != null) {
line = line.trim();
if (!line.startsWith("<!DOCTYPE ")) {
prefixBuilder.append(line + "\n");
}
}
}
prefix = prefixBuilder.toString().getBytes();
// Reset input position to just after prefix that was read.
int lines_skipped = 0;
this.in.reset();
do {
int c = this.in.read();
if (c == -1) {
break;
}
if (c == '\n') {
++lines_skipped;
}
} while (lines_skipped < lines_read);
}
This code constructs a prefix that starts with the original "<?xml" prolog
if there was one,
followed by the required POM DOCTYPE declaration. It discards any DOCTYPE
declaration
defined in the original.
The code handles the four combinations of {prolog, no-prolog} / {doctype,
no-doctype}.
For the prolog/doctype example above the begining of the output produced is:
1: <?xml version="1.0" encoding="UTF-8"?>
2: <!DOCTYPE project SYSTEM "m2-entities.ent">
3: <project xmlns="https://maven.apache.org/POM/4.0.0"; xmlns:xsi="
https://www.w3.org/2001/XMLSchema-instance";
4: xsi:schemaLocation="https://maven.apache.org/POM/4.0.0
https://maven.apache.org/xsd/maven-4.0.0.xsd";>
5:
6: <parent>
7: <groupId>com.vaadin.addon</groupId>
8: <artifactId>vaadin-charts-parent</artifactId>
9: <version>3.3.4</version>
10: </parent>
Regards
Colin Chambers
