If I do this, will this mean my site won't be indexed?
-----Original Message-----
From: A.W.F. Boer [mailto:[EMAIL PROTECTED]]
Sent: Monday, October 11, 1999 11:21 AM
To: [EMAIL PROTECTED]
Subject: Re: Real People or Search Engines?
If you do not want to give out IDs to robots, tell the robots not to visit
your servlet in your /robots.txt file. See the Web Robots FAQ at:
http://info.webcrawler.com/mak/projects/robots/faq.html
I does work for most well-behaved search engine robots.
Alexander W. F. Boer
Dept. of Computer Science and Law,
University of Amsterdam, the Netherlands
email: [EMAIL PROTECTED]
----------
Van: Lance Lavandowska[SMTP:[EMAIL PROTECTED]]
Antwoord naar: Lance Lavandowska
Verzonden: Monday, October 11, 1999 3:43 PM
Aan: [EMAIL PROTECTED]
Onderwerp: Re: Real People or Search Engines?
Search engines often/usually/sometimes (cover my bases) have a different
UserAgent header than the standard browsers. You could set up a
"pass-through" servlet to log the various UserAgents that are hitting your
site, and continue from there.
Lance Lavandowska
-----Original Message-----
From: Boemio, Neil (CAP, FGI) <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED] <[EMAIL PROTECTED]>
Date: Sunday, October 10, 1999 11:30 PM
Subject: Real People or Search Engines?
>Is there anyway to tell if my servlet is being called by a real person
>hitting my site or by search engines or robots or whatever.
>
>I ask because my servlet gives out an ID in a cookie if the user doesn't
>have one already. And it seems that I am giving out tons of IDs now!
How
>can I give out IDs to "real people" ONLY?
>
>_________________________________________________________________________
__
>To unsubscribe, send email to [EMAIL PROTECTED] and include in the
body
>of the message "signoff SERVLET-INTEREST".
>
>Archives: http://archives.java.sun.com/archives/servlet-interest.html
>Resources: http://java.sun.com/products/servlet/external-resources.html
>LISTSERV Help: http://www.lsoft.com/manuals/user/user.html
>
___________________________________________________________________________
To unsubscribe, send email to [EMAIL PROTECTED] and include in the body
of the message "signoff SERVLET-INTEREST".
Archives: http://archives.java.sun.com/archives/servlet-interest.html
Resources: http://java.sun.com/products/servlet/external-resources.html
LISTSERV Help: http://www.lsoft.com/manuals/user/user.html
___________________________________________________________________________
To unsubscribe, send email to [EMAIL PROTECTED] and include in the body
of the message "signoff SERVLET-INTEREST".
Archives: http://archives.java.sun.com/archives/servlet-interest.html
Resources: http://java.sun.com/products/servlet/external-resources.html
LISTSERV Help: http://www.lsoft.com/manuals/user/user.html
___________________________________________________________________________
To unsubscribe, send email to [EMAIL PROTECTED] and include in the body
of the message "signoff SERVLET-INTEREST".
Archives: http://archives.java.sun.com/archives/servlet-interest.html
Resources: http://java.sun.com/products/servlet/external-resources.html
LISTSERV Help: http://www.lsoft.com/manuals/user/user.html
