Re: [aseek-users] Live Updates

[Kir Kolyshkin]

Paul Stewart wrote:

When we restart indexer (runs 8 hours daily) it always reloads the
robots.txt files from what appears to be every site when it first starts
up...

That is the right behaviour, ASPseek never trust "old" robots.txt and always reloads it when starting a new indexing session. In other words,
"Period" directive doesn't apply to robots.txt.


--
== kir_at_asplinux.ru == 7551596_at_ICQ == 6722750_at_sms.beemail.ru ==

Dream like you'll live forever...Love like you've never been hurt...
Work like you don't need the money...and Dance like nobody is watching!
-- Satchel Paige