[
https://issues.apache.org/jira/browse/NUTCH-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17129531#comment-17129531
]
ASF GitHub Bot commented on NUTCH-2720:
---------------------------------------
sebastian-nagel merged pull request #528:
URL: https://github.com/apache/nutch/pull/528
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> ROBOTS metatag ignored when capitalized
> ---------------------------------------
>
> Key: NUTCH-2720
> URL: https://issues.apache.org/jira/browse/NUTCH-2720
> Project: Nutch
> Issue Type: Bug
> Components: indexer, robots
> Affects Versions: 1.15
> Reporter: Felix Zett
> Assignee: Sebastian Nagel
> Priority: Minor
> Fix For: 1.17
>
> Attachments: noindex.html
>
>
> As discussed [on the mailing
> list|https://www.mail-archive.com/[email protected]/msg16516.html],
> index-metadata fails to ignore a webpage with a capitalized robots metatag
> such as {{<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">}}. This only applies
> when parse-tika is used. parse-html will "decapitalize"
> Parsing the attached [^noindex.html] leads to the following results:
> *parse-html:*
> {code:java}
> bin/nutch parsechecker
> -Dplugin.includes="protocol-httpclient|parse-(html|metatags)|index-metadata"
> -Dindexer.delete.robots.noindex="true" -Dmetatags.names="robots"
> -Dindex.parse.md="metatag.robots" http://localhost:8080/noindex.html
> Parse Metadata: [...] metatag.robots=noindex,nofollow
> robots=noindex,nofollow{code}
> *parse-tika:*
> {code:java}
> bin/nutch parsechecker
> -Dplugin.includes="protocol-httpclient|parse-(tika|metatags)|index-metadata"
> -Dindexer.delete.robots.noindex="true" -Dmetatags.names="robots"
> -Dindex.parse.md="metatag.robots" http://localhost:8080/noindex.html
> Parse Metadata: metatag.robots=NOINDEX,NOFOLLOW [...]
> ROBOTS=NOINDEX,NOFOLLOW [...]{code}
>
> The field being named "ROBOTS" and not "robots" leads to
> {{parseData.getMeta("robots")}} being {{null}} in
> [https://github.com/apache/nutch/blob/master/src/java/org/apache/nutch/indexer/IndexerMapReduce.java#L257].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
