[
https://issues.apache.org/jira/browse/NUTCH-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193001#comment-13193001
]
Markus Jelsma commented on NUTCH-1258:
--------------------------------------
That may be a good idea indeed but we need to extend it too. This patch fixes
some issues with bad content-types but it seems the problem is bigger. The
example URL [1] doesn't provide any Content-Type in ParseMeta and a bad
Content-Type in ContentMeta, application/x-trash which is found in the HTTP
resp. header. However, parserchecker (and indexchecker) both show
contentType: text/html at the top but this value is not added to any metadata
AFAIK. In this case only contentType = content.getContentType(); returns the
desired Content-Type.
Any idea how we can get a hold on that value when we have an instance of
ParseData in the MoreIndexingFilter?
[1]: http://kam.mff.cuni.cz/conferences/GraDR/
> MoreIndexingFilter should be able to read Content-Type from both parse
> metadata and content metadata
> ----------------------------------------------------------------------------------------------------
>
> Key: NUTCH-1258
> URL: https://issues.apache.org/jira/browse/NUTCH-1258
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Affects Versions: 1.4
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
> Attachments: NUTCH-1258-1.5-1.patch
>
>
> The MoreIndexingFilter reads the Content-Type from parse metadata. However,
> this usually contains a lot of crap because web developers can set it to
> anything they like. The filter must be able to read the Content-Type field
> from content metadata as well because that contains the type detected by
> Tika's Detector.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
