AW: Regression tests was: Re: Output from NIST test suite
J. Pietschmann wrote: John Austin wrote: RedHat 9.0 (my system anyhow) includes a command 'pdftopbm' that will convert a PDF to multiple PBM (protable Bit Map) files that might be comparable. ... It would certainly help detect pixel-sized changes. That might help regression testing. We need regression tests badly. Some problems to ponder: a) Tests need to be automated for actually being useful. JUnit seems the way to go. Unfortuanately, it's still underutiliyed in FOP. b) We don't have much *unit* tests. There's only the UtilityCodeTestSuite.java. We need much more tests for basic functionality. The problem seems to be however that an elaborated test harness needs to be written in order to do unt tests for, e.g. layout managers. c) In order to test the whole engine at once, from FO input to generating PDF/whatever, well, a binary compare with a pregenerated PDF would be as sufficient as comparing bitmap images. Problems here: + The files to compare against are binary, and consume a lot of space. Well, take a look at GenericFOPTestCase.java which uses MD5 sums, one for the FO in order to detect accidental changes to the source, and one for the result. + Even small changes have potential to break the whole test suite, even if nothing important changed, let's say the order of entries in a PDF dictionary. Rendering bitmaps from PDF eliminates this, but then you wont find regressions in non-visible stuff. All in all, if there are 143 template PDFs and a change causes mismatches for all, what will you do? Examine everything, comparing pixels, check whether there are visible differences at all, and then judge whether the original or the newly generated PDF is at fault? I don't think this will be done often. Ideas welcome! J.Pietschmann As an alternative approach for c) one could create tests along the following lines: Suppose you want to test left margin properties of a block. For this a simple fo file is rendered as a bitmap. The bitmap will not be compared to a reference bitmap but some elementary assertions are calculated. For instance one such assertion could be: The rectangle of width 1 inch of the left edge is blank. I don't know of a tool that can do this but it should be pretty straight forward to implement. So, in the test suit one has a piece of fo containing a test document and some assertions in java or coded in xml that should be fulfilled by the rendered image of the fo. Assertions could contain some of the following pieces: - a specified rectangle is blank (or of some specific color) - a specified rectangle has only colors x and y (to test background and foreground colors of a block). - a given line contains the color pattern white, yellow, (white, black)*, green, white. IE. a regular expression on the colors along a line. This could be used to test border colors along a horizontal or vertical line through a bordered block. - along a given line the size of the first and last black region is at least xxx inches (to test border thickness) The advantage of this approach seems to me that it is relatively easy to maintain. The test suite is small (no binaries). It can easily be automated in junit. On the other hand, the approach is limited to relatively simple test cases. For instance it will not be possible to test font height, font style and text adjustments easily. Peter
Re: [VOTE] Finn Bock for Committer
On 24.12.2003 07:56:10 Bertrand Delacretaz wrote: Le Dimanche, 21 déc 2003, à 22:53 Europe/Zurich, Glen Mazza a écrit : ...Therefore, I'm happy to nominate Finn Bock for committer--here's my +1. Seems like no one has voted on this yet? Must be this Christmas thing... +1 from me, too. I've got an (un)fortunate additional excuse: The flu put me out of action during the last days. :-) or rather :-( BTW, I think, that when this vote passes, there are at least two other candidates who deserve a similar nomination, maybe not because of patch frequency, but for dedicated service to the project. Expect these within the next two days (no time today, sorry). Jeremias Maerki
AW: Regression tests was: Re: Output from NIST test suite
Peter Kullmann wrote: As an alternative approach for c) one could create tests along the following lines: Suppose you want to test left margin properties of a block. For this a simple fo file is rendered as a bitmap. The bitmap will not be compared to a reference bitmap but some elementary assertions are calculated. For instance one such assertion could be: The rectangle of width 1 inch of the left edge is blank. I don't know of a tool that can do this but it should be pretty straight forward to implement. There are 2 test points: renderer input and renderer output: - Renderer input: establish the SVG renderer as reference and use an 'XML file compare' program. The XSL Committee will hopefully prepare samples to demonstrate and validate concepts and rules. - Renderer output: Acrobat has a 'PDF document compare' function. This type of the tool validates the PDF renderer. The AWT renderer can be validated too by having the Java Printing System generate a PDF document with a PDF printer driver. The Java Printing System can generate PCL files as well, if better suited. There will always be problems, because equivalent, valid results may be achieved by different graphic objects or different sequences of graphic calls. Therefore there will always be automated and manual/visual methods. Hansuli Anderegg
Re: AW: Regression tests was: Re: Output from NIST test suite
On Fri, 2003-12-26 at 05:29, Peter Kullmann wrote: J. Pietschmann wrote: John Austin wrote: RedHat 9.0 (my system anyhow) includes a command 'pdftopbm' that will convert a PDF to multiple PBM (protable Bit Map) files that might be comparable. ... It would certainly help detect pixel-sized changes. That might help regression testing. I wasn't thinking of using graphics as the primary means of comparing output. It was just a thought that one could use visualization in some circumstances: + pixels that were white in both images would be rendered as white + pixels that were black in both images would be rendered as black + black pixels in the first image that were white in the second could be rendered as red + white pixels in the first image that were black in the second could be rendered as blue I thought of the idea of overlaying images for comparison when I was scrolling through the side-by-side renderings of PDF's that Finn posted yesterday (what does 'yesterday' mean in a discussion that crosses the International Date Line ?) Of course, this color-based scheme breaks down for test cases that use color. We need regression tests badly. Some problems to ponder: a) Tests need to be automated for actually being useful. JUnit seems the way to go. Unfortuanately, it's still underutiliyed in FOP. b) We don't have much *unit* tests. There's only the UtilityCodeTestSuite.java. We need much more tests for basic functionality. The problem seems to be however that an elaborated test harness needs to be written in order to do unt tests for, e.g. layout managers. c) In order to test the whole engine at once, from FO input to generating PDF/whatever, well, a binary compare with a pregenerated PDF would be as sufficient as comparing bitmap images. Problems here: + The files to compare against are binary, and consume a lot of space. Well, take a look at GenericFOPTestCase.java which uses MD5 sums, one for the FO in order to detect accidental changes to the source, and one for the result. + Even small changes have potential to break the whole test suite, even if nothing important changed, let's say the order of entries in a PDF dictionary. Rendering bitmaps from PDF eliminates this, but then you wont find regressions in non-visible stuff. All in all, if there are 143 template PDFs and a change causes mismatches for all, what will you do? Examine everything, comparing pixels, check whether there are visible differences at all, and then judge whether the original or the newly generated PDF is at fault? I don't think this will be done often. Use tests for binary equality to detect differences. Visualization might be one tool, useful in following up on detected differences. I might want to use the technique to compare the effects of changes to a document. For example: What happens on page 7 when I change space-before=10pt to space-before=15pt ? A colorized visualization would give me a better idea than separate files. Remember that our brains are all quite different. Your rote visual memory ability is probably much better than mine. You might learn more from a side-by-side comparison than I would. Crap. Now I have to give an example. Perhaps it won't take that long. Ideas welcome! J.Pietschmann As an alternative approach for c) one could create tests along the following lines: Suppose you want to test left margin properties of a block. For this a simple fo file is rendered as a bitmap. The bitmap will not be compared to a reference bitmap but some elementary assertions are calculated. For instance one such assertion could be: The rectangle of width 1 inch of the left edge is blank. I don't know of a tool that can do this but it should be pretty straight forward to implement. Probably not that hard to do once you get inside an image file in a program. Especially if you know the colors will be black (0,0,0) and white (255,255,255) or a small number of selected colors. So, in the test suit one has a piece of fo containing a test document and some assertions in java or coded in xml that should be fulfilled by the rendered image of the fo. Assertions could contain some of the following pieces: - a specified rectangle is blank (or of some specific color) - a specified rectangle has only colors x and y (to test background and foreground colors of a block). - a given line contains the color pattern white, yellow, (white, black)*, green, white. IE. a regular expression on the colors along a line. This could be used to test border colors along a horizontal or vertical line through a bordered block. - along a given line the size of the first and last black region is at least xxx inches (to test border thickness) The advantage of this approach seems to me that it is relatively
cvs commit: xml-fop/src/java/org/apache/fop/fo/extensions Label.java
gmazza 2003/12/26 14:11:17
Modified:src/java/org/apache/fop/fo FONode.java FOTreeBuilder.java
FObj.java FObjMixed.java UnknownXMLObj.java
XMLObj.java
src/java/org/apache/fop/fo/extensions Label.java
Log:
--
Bug #25646 (Patch by Finn Bock): setting SAX Locator (line and column
index of input fo stream) for debugging and better error feedback.
Revision ChangesPath
1.14 +12 -1 xml-fop/src/java/org/apache/fop/fo/FONode.java
Index: FONode.java
===
RCS file: /home/cvs/xml-fop/src/java/org/apache/fop/fo/FONode.java,v
retrieving revision 1.13
retrieving revision 1.14
diff -u -r1.13 -r1.14
--- FONode.java 16 Sep 2003 05:21:04 - 1.13
+++ FONode.java 26 Dec 2003 22:11:17 - 1.14
@@ -55,6 +55,7 @@
// XML
import org.xml.sax.Attributes;
+import org.xml.sax.Locator;
// Avalon
import org.apache.avalon.framework.logger.Logger;
@@ -91,6 +92,14 @@
}
/**
+ * Sets the name of the node.
+ * @param str the name
+ */
+public void setLocation(Locator locator) {
+// do nothing by default
+}
+
+/**
* Returns the logger for the node.
* @return the logger
*/
@@ -127,8 +136,10 @@
* @param data text
* @param start start position
* @param length length of the text
+ * @param locator location in fo source file.
*/
-protected void addCharacters(char data[], int start, int length) {
+protected void addCharacters(char data[], int start, int length,
+ Locator locator) {
// ignore
}
1.20 +14 -1 xml-fop/src/java/org/apache/fop/fo/FOTreeBuilder.java
Index: FOTreeBuilder.java
===
RCS file: /home/cvs/xml-fop/src/java/org/apache/fop/fo/FOTreeBuilder.java,v
retrieving revision 1.19
retrieving revision 1.20
diff -u -r1.19 -r1.20
--- FOTreeBuilder.java19 Sep 2003 14:33:15 - 1.19
+++ FOTreeBuilder.java26 Dec 2003 22:11:17 - 1.20
@@ -67,6 +67,7 @@
import org.apache.fop.fo.ElementMapping.Maker;
import org.apache.fop.fo.pagination.Root;
import org.xml.sax.Attributes;
+import org.xml.sax.Locator;
import org.xml.sax.SAXException;
import org.xml.sax.helpers.DefaultHandler;
@@ -110,6 +111,9 @@
/** The FOTreeControl object managing the FO Tree that is being built */
public FOTreeControl foTreeControl;
+/** The SAX locator object maneging the line and column counters */
+private Locator locator;
+
/**
* Default constructor
*/
@@ -205,12 +209,20 @@
}
/**
+ * SAX Handler for locator
+ * @see org.xml.sax.ContentHandler#setDocumentLocator(Locator)
+ */
+public void setDocumentLocator(Locator locator) {
+this.locator = locator;
+}
+
+/**
* SAX Handler for characters
* @see org.xml.sax.ContentHandler#characters(char[], int, int)
*/
public void characters(char data[], int start, int length) {
if (currentFObj != null) {
-currentFObj.addCharacters(data, start, start + length);
+currentFObj.addCharacters(data, start, start + length, locator);
}
}
@@ -264,6 +276,7 @@
try {
fobj = fobjMaker.make(currentFObj);
fobj.setName(localName);
+fobj.setLocation(locator);
fobj.handleAttrs(attlist);
} catch (FOPException e) {
throw new SAXException(e);
1.28 +15 -0 xml-fop/src/java/org/apache/fop/fo/FObj.java
Index: FObj.java
===
RCS file: /home/cvs/xml-fop/src/java/org/apache/fop/fo/FObj.java,v
retrieving revision 1.27
retrieving revision 1.28
diff -u -r1.27 -r1.28
--- FObj.java 24 Dec 2003 00:06:13 - 1.27
+++ FObj.java 26 Dec 2003 22:11:17 - 1.28
@@ -61,6 +61,7 @@
import org.apache.fop.fo.flow.Marker;
import org.apache.fop.fo.properties.FOPropertyMapping;
import org.xml.sax.Attributes;
+import org.xml.sax.Locator;
/**
* Base class for representation of formatting objects and their processing.
@@ -130,6 +131,12 @@
name = fo: + str;
}
+public void setLocation(Locator locator) {
+line = locator.getLineNumber();
+column = locator.getColumnNumber();
+systemId = locator.getSystemId();
+}
+
/**
* Handle the attributes for this element.
* The attributes must be used
DO NOT REPLY [Bug 25646] - [PATCH] line and column numbers on fo:elements
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=25646. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=25646 [PATCH] line and column numbers on fo:elements [EMAIL PROTECTED] changed: What|Removed |Added Status|NEW |RESOLVED Resolution||FIXED --- Additional Comments From [EMAIL PROTECTED] 2003-12-26 22:11 --- OK, I misunderstood--I thought the line/column numbers referred to were of the callee (say, at point where it was referencing the particular FObj), not of the FO file itself. This patch looks like a good idea, as I think it will allow us to give very solid error feedback for incorrect FO input. At any rate, if it ever becomes a performance issue, it would be fairly simple to remove, or employ a non- debugging flag as you mention. Anyway, applied, thanks! Glen
cvs commit: xml-fop/src/java/org/apache/fop/layoutmgr AddLMVisitor.java
gmazza 2003/12/26 14:39:14
Modified:src/java/org/apache/fop/layoutmgr AddLMVisitor.java
Log:
Bug 25708 (Patch submitted by Simon Pepping) -- implementation of fo:wrapper in 1.0
version.
Revision ChangesPath
1.24 +9 -3 xml-fop/src/java/org/apache/fop/layoutmgr/AddLMVisitor.java
Index: AddLMVisitor.java
===
RCS file: /home/cvs/xml-fop/src/java/org/apache/fop/layoutmgr/AddLMVisitor.java,v
retrieving revision 1.23
retrieving revision 1.24
diff -u -r1.23 -r1.24
--- AddLMVisitor.java 20 Dec 2003 06:53:23 - 1.23
+++ AddLMVisitor.java 26 Dec 2003 22:39:14 - 1.24
@@ -983,9 +983,15 @@
* @param node Wrapper object to process
*/
public void serveWrapper(Wrapper node) {
-serveFObjMixed((FObjMixed)node);
+ListIterator baseIter;
+baseIter = node.getChildren();
+if (baseIter == null) return;
+while (baseIter.hasNext()) {
+FObj child = (FObj) baseIter.next();
+child.acceptVisitor(this);
+}
}
-
+
/**
* @param node FootnoteBody object to process
*/
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
DO NOT REPLY [Bug 25708] - [PATCH] Implementation of AddLMVisitor.serveWrapper
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=25708. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=25708 [PATCH] Implementation of AddLMVisitor.serveWrapper [EMAIL PROTECTED] changed: What|Removed |Added Status|NEW |RESOLVED Resolution||FIXED --- Additional Comments From [EMAIL PROTECTED] 2003-12-26 22:50 --- Applied--thanks, Simon! It's good to see fo:wrapper now functional in 1.0.
cvs commit: xml-fop/src/java/org/apache/fop/fo BoxPropShorthandParser.java GenericShorthandParser.java Property.java PropertyList.java ShorthandParser.java
gmazza 2003/12/26 15:41:47
Modified:src/java/org/apache/fop/datatypes
ToBeImplementedProperty.java
src/java/org/apache/fop/fo BoxPropShorthandParser.java
GenericShorthandParser.java Property.java
PropertyList.java ShorthandParser.java
Log:
Property.getPropertyName() switched from returning strings to
integer constants (perh. should be renamed to getPropertyId()?);
change propagated to classes calling this function.
Revision ChangesPath
1.4 +2 -1
xml-fop/src/java/org/apache/fop/datatypes/ToBeImplementedProperty.java
Index: ToBeImplementedProperty.java
===
RCS file:
/home/cvs/xml-fop/src/java/org/apache/fop/datatypes/ToBeImplementedProperty.java,v
retrieving revision 1.3
retrieving revision 1.4
diff -u -r1.3 -r1.4
--- ToBeImplementedProperty.java 22 Dec 2003 03:53:31 - 1.3
+++ ToBeImplementedProperty.java 26 Dec 2003 23:41:47 - 1.4
@@ -67,6 +67,7 @@
if (p instanceof ToBeImplementedProperty) {
return p;
}
+
ToBeImplementedProperty val =
new ToBeImplementedProperty(getPropName());
return val;
@@ -77,7 +78,7 @@
* Constructor
* @param propName name of Property
*/
-public ToBeImplementedProperty(String propName) {
+public ToBeImplementedProperty(int propId) {
//XXX: ([EMAIL PROTECTED]) This is a bit of a kluge, perhaps an
//UnimplementedPropertyException or something similar should
1.4 +7 -5 xml-fop/src/java/org/apache/fop/fo/BoxPropShorthandParser.java
Index: BoxPropShorthandParser.java
===
RCS file: /home/cvs/xml-fop/src/java/org/apache/fop/fo/BoxPropShorthandParser.java,v
retrieving revision 1.3
retrieving revision 1.4
diff -u -r1.3 -r1.4
--- BoxPropShorthandParser.java 17 Jul 2003 17:31:28 - 1.3
+++ BoxPropShorthandParser.java 26 Dec 2003 23:41:47 - 1.4
@@ -49,6 +49,7 @@
* Software Foundation, please see http://www.apache.org/.
*/
package org.apache.fop.fo;
+import org.apache.fop.fo.properties.FOPropertyMapping;
/**
* Shorthand property parser for Box properties
@@ -69,17 +70,18 @@
* @see org.apache.fop.fo.GenericShorthandParser#convertValueForProperty(String,
* Property.Maker, PropertyList)
*/
-protected Property convertValueForProperty(String propName,
+protected Property convertValueForProperty(int propId,
Property.Maker maker,
PropertyList propertyList) {
+String name = FOPropertyMapping.getPropertyName(propId);
Property p = null;
-if (propName.indexOf(-top) = 0) {
+if (name.indexOf(-top) = 0) {
p = getElement(0);
-} else if (propName.indexOf(-right) = 0) {
+} else if (name.indexOf(-right) = 0) {
p = getElement(count() 1 ? 1 : 0);
-} else if (propName.indexOf(-bottom) = 0) {
+} else if (name.indexOf(-bottom) = 0) {
p = getElement(count() 2 ? 2 : 0);
-} else if (propName.indexOf(-left) = 0) {
+} else if (name.indexOf(-left) = 0) {
p = getElement(count() 3 ? 3 : (count() 1 ? 1 : 0));
}
// if p not null, try to convert it to a value of the correct type
1.3 +9 -7 xml-fop/src/java/org/apache/fop/fo/GenericShorthandParser.java
Index: GenericShorthandParser.java
===
RCS file: /home/cvs/xml-fop/src/java/org/apache/fop/fo/GenericShorthandParser.java,v
retrieving revision 1.2
retrieving revision 1.3
diff -u -r1.2 -r1.3
--- GenericShorthandParser.java 1 Sep 2003 18:33:05 - 1.2
+++ GenericShorthandParser.java 26 Dec 2003 23:41:47 - 1.3
@@ -52,6 +52,7 @@
import java.util.Vector;
import java.util.Enumeration;
+import org.apache.fop.fo.properties.FOPropertyMapping;
public class GenericShorthandParser implements ShorthandParser {
@@ -72,7 +73,7 @@
*/
protected Property getElement(int index) {
if (list.size() index) {
-return (Property)list.elementAt(index);
+return (Property) list.elementAt(index);
} else {
return null;
}
@@ -87,7 +88,7 @@
// Stores 1 to 3 values for border width, style, color
// Used for: border, border-top, border-right etc
-public Property getValueForProperty(String propName,
+public Property getValueForProperty(int propId,
