Sorry about being so vague. I was trying to run Fetcher (and Fetcher2
as well) and noticed that every fetched page was throwing an NPE.
Essentially, the NPE was coming from Content.java that referenced
HttpResponse and the line that was causing the problem had a call to
MimeUtils.getRepository(). The snippet of code looks like this:
MimeType type = null;
String cleanedMimeType = null;
try {
cleanedMimeType = MimeType.clean(typeName);
} catch (MimeTypeException mte) {
// Seems to be a malformed mime type name...
}
// first try to get the type from the cleaned type name
type = cleanedMimeType != null ? this.mimeTypes.getRepository().forName(
cleanedMimeType) : null;
// if returned null, then try url resolution
if (type == null) {
// If no mime-type header, or cannot find a corresponding registered
// mime-type, then guess a mime-type from the url pattern
type = this.mimeTypes.getRepository().getMimeType(url) != null ?
this.mimeTypes
.getRepository().getMimeType(url)
: type;
}
// if magic is enabled use mime magic to guess if the mime type returned
// from the magic guess is different than the one that's already set
so far
// if it is, go with the mime type returned by the magic
if (this.mimeTypeMagic) {
MimeType magicType = this.mimeTypes.getRepository().getMimeType(data);
if (magicType != null &&
!type.getName().equals(magicType.getName())) {
// If magic enabled and the current mime type differs from that
of the
// one returned from the magic, take the magic mimeType
type = magicType;
}
}
return type.getName();
}
The NPE was being thrown on the last line, so I did some tracing and
found out that the call to MimeType.clean(typeName) [typeName <-
"text/html] worked fine, but the next line caused a problem. The
this.mimeTypes.getRepository.forName(cleanedMimeType) was returning
null. My problem was that I downloaded the trunk and it didn't have a
MimeUtils anymore so I had no way to trace this.
Anyway, after an hour or so of banging my head against the wail I
realized the update to Nutch didn't have the correct .xml file
describing mime types in the conf/ directory. Thus, I unzipped the Tika
jar, grabbed the .xml file and changed nutch-default.xml to point to
that xml for mime types and it started working.
Sorry again for being so vague. I'm not sure if I should submit a JIRA
issue for this, but I'm happy to do so if anyone else has seen this issue.
Thanks,
Ned
Chris Mattmann wrote:
Hi Ned,
Glad to see you're poking around with the Tika software and its use in
Nutch. To start, you probably want to go to the website for Tika:
http://incubator.apache.org/tika/
On that website, you should see the links to the SVN repository. The
version of Tika that was used was a version that I built the same day I
committed the fix for NUTCH-562:
http://issues.apache.org/jira/browse/NUTCH-562
Which appears to be a version of Tika built on October 8th. The API for the
mime framework has changed a bit since then (to its betterment), however, I
neglected to upgrade the Nutch API because of the strong objection I
received from Andrzej and input from Dennis Kubes regarding the use of the
Tika API in Nutch. I stand by my email I sent in reply to the objections:
http://www.nabble.com/forum/ViewPost.jtp?post=13142174&framed=y
However, out of respect for the other committers, neglected to make any
updates to the Nutch use of the Tika API since I never heard back from
anyone after my response.
That said, could you be a bit more specific Ned as to the exact problem
you're having, e.g., "I tried visiting this site (URL here), the content
type was (content/type here), and then it got into Content.java, and on line
XXX it seems that the MimeType is getting set to null when it tries to...".
With that info, I could probably help you quite a bit more. Also, depending
upon how the rest of the Nutch committers want to handle the use of Tika
(revert and remain stagnant, or use Tika and leverage the updates we're
making to the Mime framework there), then we could come up with a strategy
to help you out with the issue you're having.
Thanks!
Cheers,
Chris
On 11/6/07 3:47 PM, "Ned Rockson" <[EMAIL PROTECTED]> wrote:
I think there may be a bug in the Content.java when it tries to convert
the textual representation of the type to a MimeType. It always returns
null. I'm trying to fix it but I can't find an API for Tika (or even
src). Can someone point me in the right direction?
Thanks,
Ned
______________________________________________
Chris Mattmann, Ph.D.
[EMAIL PROTECTED]
_________________________________________________
Jet Propulsion Laboratory Pasadena, CA
Office: 171-266B Mailstop: 171-246
_______________________________________________________
Disclaimer: The opinions presented within are my own and do not reflect
those of either NASA, JPL, or the California Institute of Technology.
