[ 
https://issues.apache.org/jira/browse/NUTCH-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16846773#comment-16846773
 ] 

Felix Zett commented on NUTCH-2720:
-----------------------------------

Changing the line mentioned to

{{String robotsMeta = parseData.getMeta("metatag.robots");}}

fixes the problem, I don't know if it's the right way to go though.

> ROBOTS metatag ignored when capitalized
> ---------------------------------------
>
>                 Key: NUTCH-2720
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2720
>             Project: Nutch
>          Issue Type: Bug
>          Components: indexer, robots
>    Affects Versions: 1.15
>            Reporter: Felix Zett
>            Priority: Minor
>         Attachments: noindex.html
>
>
> As discussed [on the mailing 
> list|https://www.mail-archive.com/user@nutch.apache.org/msg16516.html], 
> index-metadata fails to ignore a webpage with a capitalized robots metatag 
> such as {{<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">}}. This only applies 
> when parse-tika is used. parse-html will "decapitalize"
> Parsing the attached [^noindex.html] leads to the following results:
> *parse-html:*
> {code:java}
> bin/nutch parsechecker 
> -Dplugin.includes="protocol-httpclient|parse-(html|metatags)|index-metadata" 
> -Dindexer.delete.robots.noindex="true" -Dmetatags.names="robots" 
> -Dindex.parse.md="metatag.robots" http://localhost:8080/noindex.html
> Parse Metadata: [...] metatag.robots=noindex,nofollow 
> robots=noindex,nofollow{code}
> *parse-tika:*
> {code:java}
> bin/nutch parsechecker 
> -Dplugin.includes="protocol-httpclient|parse-(tika|metatags)|index-metadata" 
> -Dindexer.delete.robots.noindex="true" -Dmetatags.names="robots" 
> -Dindex.parse.md="metatag.robots" http://localhost:8080/noindex.html
> Parse Metadata: metatag.robots=NOINDEX,NOFOLLOW  [...] 
> ROBOTS=NOINDEX,NOFOLLOW [...]{code}
>  
> The field being named "ROBOTS" and not "robots" leads to 
> {{parseData.getMeta("robots")}} being {{null}} in 
> [https://github.com/apache/nutch/blob/master/src/java/org/apache/nutch/indexer/IndexerMapReduce.java#L257].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to