> I think this one is a keeper, unfortunately. If someone comes up with a
> better way to keep search engines from being evil, I'm willing to work
> on it, but I don't see any way to do a POST request from a textual link.
I'm not sure about this, but what about making a pseudo directory for the
buttons.
That is, instead of the URL of the history page being
http://mysite/myswiki/2.history
it could by
http://mysite/myswiki/foo/2.history
My understanding of how a webcrawler should work is that before it reads
any files in directory foo, it will first try to get foo/robots.txt. If
this file exists, it wouldn't index that directory.
So, for pages that you don't want to be indexed, you make their URL's
pointers to "virtual directories."
Eg:
http://mysite/myswiki/1 remains the same, but
http://mysite/myswiki/1.edit becomes
http://mysite/myswiki/virtual/1.edit
and then virtual/robots.txt would return some content to avoid those links
being chased.
I don't know if that would work, though. Any thoughts?
-Eric
