- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: Maxime Subject: Re: Duplicated crawling
Yes, dataparksearch take a reference from db before crawling, however it can crawl same URL several times with different Accept-Language header specified according VaryLang command. Also, the indexer crawls again all expired documets, i.e. all documents have been indexed more time period ago that specified by Period command. You may prevent such reindexing by specifying very high value for Period command. - - - - - - - - - - - - - - - - - - - - - - - - - - - - Read the full topic here: http://www.dataparksearch.org/cgi-bin/simpleforum.cgi?fid=02;topic_id=1172116522
