Author: Alexander Barkov
> > Thanks for your quick answer.
> > I tried to add the NoIndexIf but i cannot get it to work.
> > I used the indexer.conf default file, and added the two following lines at
> > the end of that file :
> > Server http://www.wearethelous.com/feed/
> > NoIndexIf Content-Type application/rss+xml
> I tried the same thing, and it seems to work fine.
> This page is not returned in search results.
> If I remove the NoIndexIf command, this page IS returned by search results.
> Note, indexer shows the URL in its log, because it still must
> download this URL to know its content type.
> But the fact that you can see the "SectionFilter:..." line in the log
> tells that indexer marks it as "not for indexing" and thus stores no data
> into the underlying tables cachedcopy and bdicti, so "indexer --index" later
> does see it when creating the search index.
Note, if you know that documents under certain location return
application/rss+xml or some other not desired content type,
then consider using Disallow instead. In this case indexer will
not even download these documents.
NoIndexIf is rather for the cases when it's not possible to describe "bad"
documents by their URL pattern.
General mailing list