What is the best way to cloak a site - send search engines different
content than real users?
Yes, I know it's bad practice, and I know the domain will eventually
be banned. I've found lots of different methods including huge tables
of all the possible client types sent by various spiders.
On Aug 11, 2005, at 3:44 PM, Evert | Collab wrote:
Use robots.txt
'evil' searchengines will spoof the user-agent string anyway
Can you be more specific about what you mean by use robots.txt?
I just want to cloak for Google, MSN, and Yahoo. I couldn't care less
about what any other search
On Aug 11, 2005, at 4:06 PM, Evert | Collab wrote:
First hit on google:
http://www.searchengineworld.com/robots/robots_tutorial.htm
Search engines check for a robots.txt on your site, in the
robots.txt file you can specify that certain or all search engines
shouldn't index your site
I
Brian Dunning wrote:
On Aug 11, 2005, at 3:44 PM, Evert | Collab wrote:
Use robots.txt
'evil' searchengines will spoof the user-agent string anyway
Can you be more specific about what you mean by use robots.txt?
I just want to cloak for Google, MSN, and Yahoo. I couldn't care less
about
Jasper Bryant-Greene wrote:
Brian Dunning wrote:
On Aug 11, 2005, at 3:44 PM, Evert | Collab wrote:
Use robots.txt
'evil' searchengines will spoof the user-agent string anyway
Can you be more specific about what you mean by use robots.txt?
I just want to cloak for Google, MSN, and
Jochem Maas wrote:
Jasper Bryant-Greene wrote:
robots.txt will not do what you want it to.
Just sniff for those robots' User-Agents (Google, MSN and Yahoo all
publish their UA strings on their websites, AFAIK) and send different
content if it's one of those.
they will hammer you for it
robots.txt will not do what you want it to.
Just sniff for those robots' User-Agents (Google, MSN and Yahoo all publish
their UA strings on their websites, AFAIK) and send different content if
it's one of those.
they will hammer you for it eventually - AFAICT all major SEs send out their
Philip Hallstrom wrote:
robots.txt will not do what you want it to.
Just sniff for those robots' User-Agents (Google, MSN and Yahoo all
publish their UA strings on their websites, AFAIK) and send different
content if it's one of those.
they will hammer you for it eventually - AFAICT all
* Brian Dunning [EMAIL PROTECTED] :
On Aug 11, 2005, at 4:06 PM, Evert | Collab wrote:
First hit on google:
http://www.searchengineworld.com/robots/robots_tutorial.htm
Search engines check for a robots.txt on your site, in the
robots.txt file you can specify that certain or all search
Evert | Collab wrote:
Lets just put it this way:
if you don't want your site indexed, use robots.txt
if you want to hide your site from search engines [ which won't even
touch your files if you use robots.txt ] check the UA string.
I can't imagine a situation where you want to hide your
Jasper Bryant-Greene wrote:
Jochem Maas wrote:
Jasper Bryant-Greene wrote:
robots.txt will not do what you want it to.
Just sniff for those robots' User-Agents (Google, MSN and Yahoo all
publish their UA strings on their websites, AFAIK) and send different
content if it's one of those.
11 matches
Mail list logo
