- - - - - - - - - - - - - - - - - - - - - - - - - - - - Name: Maxime Subject: Re: bugs in indexer (crawling)
"Server world" command means any URL in the world. Probably you shouldn't use "world" option with your Server commands. What do you expect to get with your config ? robots.txt are special files, they are requested despite of absance of Realm/Server command. If any URL from a site is requested, then the robots.txt file is requested too from that site. - - - - - - - - - - - - - - - - - - - - - - - - - - - - Read the full topic here: http://www.dataparksearch.org./cgi-bin/simpleforum.cgi?fid=03;topic_id=1228476957
