Hi Felix, > There are plenty of resources online [3] suggesting a capitalized ROBOTS meta > tag, > and I can't seem to find any that say that it MUST be in lower case. > So I guess this can still be considered a bug.
Yes, definitely. Please open a Jira issue to fix it. Thanks, Sebastian On 5/23/19 3:16 PM, Felix von Zadow wrote: > Hi Sebastian, > > thank you so much for checking again. With your test document I get the same > result as you. Guess what the difference to my document was... Mine has the > robots tag capitalized: > <META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW"> > Apparently someone thought this tag was particularly important. Or they came > from the 90s where this was common practice [1]. > > Anyway, this leads to the following results: > > parse-html: > bin/nutch parsechecker > -Dplugin.includes="protocol-httpclient|parse-(html|metatags)|index-metadata" > -Dmetatags.names="robots" -Dindex.parse.md="metatag.robots" > http://localhost:8080/noindex_caps.html > Parse Metadata: CharEncodingForConversion=windows-1252 > OriginalCharEncoding=windows-1252 metatag.robots=noindex,nofollow > robots=noindex,nofollow > > parse-tika: > bin/nutch parsechecker > -Dplugin.includes="protocol-httpclient|parse-(tika|metatags)|index-metadata" > -Dmetatags.names="robots" -Dindex.parse.md="metatag.robots" > http://localhost:8080/noindex_caps.html > Parse Metadata: metatag.robots=NOINDEX,NOFOLLOW dc:title=test > Content-Encoding=ISO-8859-1 ROBOTS=NOINDEX,NOFOLLOW Content-Type=text/html; > charset=ISO-8859-1 > > Note that parse-tika keeps the capitalization of "ROBOTS" while parse-html > does not. So my guess is that in [2] parseData.getMeta("robots") becomes null > and then the document is indexed. > > There are plenty of resources online [3] suggesting a capitalized ROBOTS meta > tag, and I can't seem to find any that say that it MUST be in lower case. So > I guess this can still be considered a bug. > > Felix > > > [1] https://www.w3.org/TR/html401/appendix/notes.html#h-B.4.1.2 > [2] > https://github.com/apache/nutch/blob/master/src/java/org/apache/nutch/indexer/IndexerMapReduce.java#L257 > [3] http://www.robotstxt.org/meta.html > >> Von: Sebastian Nagel >> >> Hi Felix, >> >> I've also checked parse-tika but the "robots=noindex" is in the parse >> metadata >> also when, >> at least, for the following test document: >> >> % cat /var/www/html/nutch/noindex.html >> <html> >> <head> >> <title>test</title> >> <meta name='robots' content='noindex,nofollow'> >> </head> >> <body> >> test for robots=noindex >> </body> >> </html> >> >> The test page is hosted via Apache httpd on >> http://localhost/nutch/noindex.html: >> >> - using parse-html >> >> % bin/nutch parsechecker \ >> -Dplugin.includes="protocol-httpclient|parse-(html|metatags)|index- >> metadata" \ >> -Dmetatags.names="robots" -Dindex.parse.md="metatag.robots" \ >> http://localhost/nutch/noindex.html >> ... >> Parse Metadata: CharEncodingForConversion=windows-1252 >> OriginalCharEncoding=windows-1252 >> metatag.robots=noindex,nofollow metatag.robots=noindex,nofollow >> robots=noindex,nofollow >> >> - using parse-tika >> >> % bin/nutch parsechecker \ >> >> -Dplugin.includes="protocol-httpclient|parse-(tika|metatags)|index-metadata" >> \ >> -Dmetatags.names="robots" -Dindex.parse.md="metatag.robots" \ >> http://localhost/nutch/noindex.html >> ... >> Parse Metadata: metatag.robots=noindex,nofollow >> metatag.robots=noindex,nofollow dc:title=test >> Content-Encoding=ISO-8859-1 robots=noindex,nofollow Content- >> Type=text/html; charset=ISO-8859-1 >> >> >> But I believe you. There can multiple other reasons. Could you share the >> HTML or >> a snippet of it >> which makes the issue reproducible? >> >> >> Thanks, >> Sebastian >> >> On 5/23/19 10:01 AM, Felix von Zadow wrote: >>> >>> Hi Sebastian, >>> >>> thank you for trying to reproduce the problem! >>> >>>> The parse-metatags plugin only duplicates the "robots" metatags, >>>> adding it also as "metatag.robots" but keep the original "robots". >>> >>> This got me confused for a minute because you are absolutely right, they're >> both there. So I checked again to see how I got to a point where I only had >> "metatag.robots" but not "robots" and the problem only seems to occur when >> parse-tika is used instead of parse-html. So with this minimal setup >>> >>> protocol-httpclient|parse-(tika|metatags)|index-(metadata) >>> >>> the parse metadata only contains "metatag.robots" while with this setup >>> >>> protocol-httpclient|parse-(html|metatags)|index-(metadata) >>> >>> the parse metadata contains both "metatag.robots" and "robots". >>> >>> Felix >>> >>> >>>> Von: Sebastian Nagel >>>> >>>> Hi Felix, >>>> >>>> I tried to reproduce the problem. The parse-metatags plugin only duplicates >> the >>>> "robots" metatags, >>>> adding it also as "metatag.robots" but keep the original "robots". >>>> >>>> That is the case using the current master: >>>> >>>> - with parse-metatags and metatags.names="robots" the ParseData object >>>> contains: >>>> >>>> Parse Metadata: CharEncodingForConversion=utf-8 >> OriginalCharEncoding=utf-8 >>>> metatag.robots=noindex,nofollow metatag.robots=noindex,nofollow >>>> generator=WordPress 3.1 >>>> robots=noindex,nofollow >>>> >>>> metatag.robots is even added twice, but most important "robots" is still >> present >>>> >>>> - without: >>>> >>>> Parse Metadata: CharEncodingForConversion=utf-8 >> OriginalCharEncoding=utf-8 >>>> generator=WordPress 3.1 >>>> robots=noindex,nofollow >>>> >>>> >>>> Deleting the robots=noindex documents works as expected for both settings. >>>> >>>>> Or is it and I should file a report/patch? >>>> >>>> Yes, please open an issue to fix this on >>>> https://issues.apache.org/jira/projects/NUTCH >>>> >>>> Could be that there is some additional condition which I didn't hit. >>>> >>>> Can you also share the document for which it does not work it does not >>>> work? >>>> >>>> >>>> Thanks, >>>> Sebastian >>>> >>>> >>>> >>>> On 5/13/19 11:34 AM, Felix von Zadow wrote: >>>>> Hi all! >>>>> >>>>> So I was trying to use the option indexer.delete.robots.noindex (exclude >> page >>>> when <meta robots="noindex"> is encountered). >>>>> >>>>> However, the page I'm testing with is still being indexed. I have parse- >> metatags >>>> and index-metadata activated and indexer.delete.robots.noindex=true, >>>> metatags.names="robots" and index.parse.md="metatag.robots". >>>>> >>>>> Looking at IndexerMapReduce.java (#257) [1], the field that is being >>>>> checked >> is >>>> "robots" and not "metatag.robots". It does work as expected when I change >>>> it >> to >>>> "metatag.robots": >>>>> >>>>> Before: >>>>> Indexing 3/3 documents >>>>> Deleting 0 documents >>>>> Indexer: number of documents indexed, deleted, or skipped: >>>>> Indexer: 3 indexed (add/update) >>>>> >>>>> After: >>>>> Indexing 2/2 documents >>>>> Deleting 0 documents >>>>> Indexer: number of documents indexed, deleted, or skipped: >>>>> Indexer: 1 deleted (robots=noindex) >>>>> Indexer: 2 indexed (add/update) >>>>> >>>>> >>>>> Am I missing something and this is not actually a bug but rather some >>>> misconfiguration on my part? >>>>> Or is it and I should file a report/patch? >>>>> >>>>> Thanks! >>>>> Felix >>>>> >>>>> >>>>> [1] >>>> >> https://github.com/apache/nutch/blob/master/src/java/org/apache/nutch/inde >>>> xer/IndexerMapReduce.java#L257 >>>>> >>>>> >>> >
