Hi Iain,
> I have copied tika-mimetypes.xml from the tika jar file and installed a copy
> in my configuration directory. I have updated nutch-site.xml to point to
> this file and the log entries indicate that this is being found.
... and the property mime.type.magic is true (default)?
> <mime-type type="application/json">
> <sub-class-of type="application/javascript"/>
Just as a trial: What happens if you make the web server return
"application/javascript"
as content type?
> I am still getting the content type detected as text/html and the json
> parser is not being invoked. Any suggestions as to what to look at next?
The mime magic is done by Tika. Nutch (o.a.n.util.MimeUtil) passes the following
resources to Tika:
- byte stream for magic detection
- URL for additional file name patterns
- content type sent by server
URL and server content type are required as additional hints, e.g.,
for zip containers such as .xlsx, etc.
I fear that you have to run a debugger to find out what is going wrong.
I would also run first Tika alone with the modified tika-mimetypes.xml,
just to make sure that the mime magic works as expected.
Cheers,
Sebastian
On 04/13/2015 04:26 PM, Iain Lopata wrote:
> I have a page that I am fetching that contains JSON and I have a plugin for
> parsing JSON.
>
>
>
> The server sets a mimetype of "text/html" and consequently my json parser
> does not get invoked.
>
>
>
> If I run parsechecker from the command line and specify -forceAs
> "application/json" the json parser is invoked and works successfully.
>
>
>
> So, I believe that if I can get tika to give me "application/json" as the
> detected content type for this page, it should work during a crawl.
>
>
>
> I have copied tika-mimetypes.xml from the tika jar file and installed a copy
> in my configuration directory. I have updated nutch-site.xml to point to
> this file and the log entries indicate that this is being found.
>
>
>
> In my copy of tika-mimetypes.xml I have added the match rule shown below
>
>
>
> <mime-type type="application/json">
>
> <sub-class-of type="application/javascript"/>
>
> <magic priority="100">
>
> <match value="{" type="string" offset="0"/>
>
> </magic>
>
> <glob pattern="*.json"/>
>
> </mime-type>
>
>
>
> I know that my match is much too broad, but I am using this just while
> trying to resolve this problem.
>
>
>
> I have also set lang.extraction.policy to identify in nutch-site.xml (again
> primarily for testing purposes).
>
>
>
> I am still getting the content type detected as text/html and the json
> parser is not being invoked. Any suggestions as to what to look at next?
>
>
>
> Thanks!
>
>
>
> Iain
>
>
