please find attached a bug fix that fix the problem described by Massimo Miccoli.
The problem:
A http header does not provides a content-type information.
In the new Conten.java the field content type was set to null.
Until serializing the conent object with a null field the null-pointer exception occurred.
My solution:
1.) When there is no content-type information we set the content type to "unknown" and not to null.
2.) the html parser checks if the content type is not unknown. In case it is not uknown the html parser checks if it the content type is text/html.
3.) I set the url suffix for the html parser from nothing to "html" in the plugin.xml.
Further suggestions:
I would allow multiple suffixes in the plugin.xml for one content parser.
Now:
<x-tad-bigger><implementation id="net.nutch.parse.html.HtmlParser"
class="net.nutch.parse.html.HtmlParser"
contentType="text/html"
pathSuffix="html"/>
Suggestion:
<implementation id="net.nutch.parse.html.HtmlParser"
class="net.nutch.parse.html.HtmlParser"
contentType="text/html"
pathSuffix="html, htm, jsp, php, asp"/></x-tad-bigger>
To get this working we need to change as well one line in the ParserFactory:
(line 103)
Now:
<x-tad-bigger>if</x-tad-bigger><x-tad-bigger> (suffix.equals(extension.getAttribute(</x-tad-bigger><x-tad-bigger>"pathSuffix"</x-tad-bigger><x-tad-bigger>)))
Suggestion:
</x-tad-bigger><x-tad-bigger>if</x-tad-bigger><x-tad-bigger> (extension.getAttribute(</x-tad-bigger><x-tad-bigger>"pathSuffix"</x-tad-bigger><x-tad-bigger>).indexOf(suffix)=>0)</x-tad-bigger>
Cheers,
Stefan
Index: src/java/net/nutch/protocol/Content.java
===================================================================
RCS file: /cvsroot/nutch/nutch/src/java/net/nutch/protocol/Content.java,v
retrieving revision 1.1
diff -u -r1.1 Content.java
--- src/java/net/nutch/protocol/Content.java 1 Jun 2004 20:12:39 -0000 1.1
+++ src/java/net/nutch/protocol/Content.java 4 Jun 2004 22:10:09 -0000
@@ -27,7 +27,7 @@
this.url = url;
this.base = base;
this.content = content;
- this.contentType = contentType;
+ setContentType(contentType);
this.metaData = metaData;
}
@@ -96,6 +96,9 @@
*/
public String getContentType() { return contentType; }
public void setContentType(String contentType) {
+ if(contentType==null){
+ contentType = "unknown";
+ }
this.contentType = contentType;
}
Index: src/plugin/parse-html/plugin.xml
===================================================================
RCS file: /cvsroot/nutch/nutch/src/plugin/parse-html/plugin.xml,v
retrieving revision 1.2
diff -u -r1.2 plugin.xml
--- src/plugin/parse-html/plugin.xml 4 Jun 2004 20:24:54 -0000 1.2
+++ src/plugin/parse-html/plugin.xml 4 Jun 2004 22:10:09 -0000
@@ -27,7 +27,7 @@
<implementation id="net.nutch.parse.html.HtmlParser"
class="net.nutch.parse.html.HtmlParser"
contentType="text/html"
- pathSuffix=""/>
+ pathSuffix="html"/>
</extension>
Index: src/plugin/parse-html/src/java/net/nutch/parse/html/HtmlParser.java
===================================================================
RCS file:
/cvsroot/nutch/nutch/src/plugin/parse-html/src/java/net/nutch/parse/html/HtmlParser.java,v
retrieving revision 1.2
diff -u -r1.2 HtmlParser.java
--- src/plugin/parse-html/src/java/net/nutch/parse/html/HtmlParser.java 4 Jun 2004
20:24:54 -0000 1.2
+++ src/plugin/parse-html/src/java/net/nutch/parse/html/HtmlParser.java 4 Jun 2004
22:10:10 -0000
@@ -15,10 +15,10 @@
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;
import org.w3c.dom.*;
-import org.w3c.dom.html.*;
+
import org.apache.html.dom.*;
-import net.nutch.fetcher.FetcherOutput;
+
import net.nutch.protocol.Content;
import net.nutch.util.*;
import net.nutch.parse.*;
@@ -47,7 +47,7 @@
// check that contentType is one we can handle
String contentType = content.getContentType();
- if (contentType != null && !contentType.startsWith("text/html"))
+ if (!contentType.startsWith("unknown") && !contentType.startsWith("text/html"))
throw new ParseException("Content-Type not text/html: " + contentType);
// parse the content
