Hi,
Googlebot and msnbot are supposed to obey robots.txt, but they are ignoring my
robots.txt ( http://simpy.com/robots.txt ), that contains:
User-agent: *
Disallow: /simpy/
It's been more than 2 weeks since I've updated my robots.txt, yet I still see
this from Googlebot and msnbot:
66.249.65
On Mar 26, 2006, at 7:25 AM, <[EMAIL PROTECTED]> [EMAIL PROTECTED]> wrote:
Hi,
Googlebot and msnbot are supposed to obey robots.txt, but they are
ignoring my robots.txt ( http://simpy.com/robots.txt ), that contains:
Looks like a bug to me -Tim
___
--On March 26, 2006 7:25:42 AM -0800 [EMAIL PROTECTED] wrote:
>
> Googlebot and msnbot are supposed to obey robots.txt, but they are ignoring
> my robots.txt ( http://simpy.com/robots.txt ), that contains:
>
> User-agent: *
> Disallow: /simpy/
>
> User-agent: Googlebot
> Disallow: /rss/
You need
Ah, thanks for pointing this out!
I did run the robots.txt validator, and it said I had everything done
correctly, but apparently I don't.
Thanks again! Fixed it - http://simpy.com/robots.txt
Otis
- Original Message
From: Walter Underwood <[EMAIL PROTECTED]>
To: "Internet robots, spide
They should stop crawling immediately. They are supposed to look at
that file before they crawl your site.
Years ago, Scooter used to plague me daily. As soon as I put that
robots file blocking him, he stopped.
Regards,
F
I just looked at your file.
You have the wildcard at the top. You need to move 'User-agent: *,
Disallow: /simpy/' to the end of the file. It should be the very last
entry.
What happens is that Googlebot gets to the * and accepts the
instructions there. It never gets to its own indi
--On March 26, 2006 2:16:13 PM -0500 Fred Atkinson <[EMAIL PROTECTED]> wrote:
>
> You have the wildcard at the top. You need to move 'User-agent: *,
> Disallow: /simpy/' to the end of the file. It should be the very last
> entry.
>
> What happens is that Googlebot gets to the * and accep
Thanks to both of you. I've adopted Fred's suggestion, even though the order
should have no significance. It shouldn't hurt. If both of your suggestions
don't help, I'll follow up here.
Otis
- Original Message
From: Walter Underwood <[EMAIL PROTECTED]>
To: "Internet robots, spiders,
>This is wrong.
>
>The spec [1] doesn't say anything about order being significant. One of the
>examples in the spec shows the robot "cybermapper" matching a user-agent
>line which is after a "*" entry.
>
>A robot which implements "first match" is not following the spec.
>
>The spec text in the pro
