I am asking a few websites to allow me to index there site, what you they add to the robots.txt and where do i get the exact name of my crawler.

Cheers.
Shane

Reply via email to