[
https://issues.apache.org/jira/browse/NUTCH-1075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082610#comment-13082610
]
Julien Nioche commented on NUTCH-1075:
--------------------------------------
Hi Lewis,
One way of testing would be to call o.a.n.parse.ParserChecker on some documents
and make sure that the language metadata has been set property. Otherwise
having some test classes would be good as well.
This will make our code a bit lighter and give us more flexibility as we'll
have more choice in the strategies to adopt i.e. extract vs identify
Thanks
Julien
> Delegate language identification to Tika
> ----------------------------------------
>
> Key: NUTCH-1075
> URL: https://issues.apache.org/jira/browse/NUTCH-1075
> Project: Nutch
> Issue Type: Improvement
> Components: parser
> Affects Versions: 1.4
> Reporter: Julien Nioche
> Assignee: Julien Nioche
> Fix For: 1.4
>
> Attachments: NUTCH-1075.patch
>
>
> In 2.0 the language identification is delegated to Tika and is done as part
> of the parsing step (and not during the indexing as done currently).
> The patch attached is a backport from trunk which implements this and adds a
> new parameter to determine the strategy to use
> {code:xml}
> <property>
> <name>lang.extraction.policy</name>
> <value>detect,identify</value>
> <description>This determines when the plugin uses detection and
> statistical identification mechanisms. The order in which the
> detect and identify are written will determine the extraction
> policy. Default case (detect,identify) means the plugin will
> first try to extract language info from page headers and metadata,
> if this is not successful it will try using tika language
> identification. Possible values are:
> detect
> identify
> detect,identify
> identify,detect
> </description>
> </property>
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
