No and yes, depending on what you specify. In general indexing a servlet isn't
likely to be useful (unless it's purely a filtering servlet). So simply
disallow robots from your servlet paths e.g.
User-agent: *
Disallow: /servlets
in your /robots.txt file
If your site is all servlets then that might not be such a good idea.
- simon
"Boemio, Neil (CAP, FGI)" wrote:
> If I do this, will this mean my site won't be indexed?
>
> -----Original Message-----
> From: A.W.F. Boer [mailto:[EMAIL PROTECTED]]
> Sent: Monday, October 11, 1999 11:21 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Real People or Search Engines?
>
> If you do not want to give out IDs to robots, tell the robots not to visit
> your servlet in your /robots.txt file. See the Web Robots FAQ at:
>
> http://info.webcrawler.com/mak/projects/robots/faq.html
>
> I does work for most well-behaved search engine robots.
>
> Alexander W. F. Boer
> Dept. of Computer Science and Law,
> University of Amsterdam, the Netherlands
> email: [EMAIL PROTECTED]
>
> ----------
> Van: Lance Lavandowska[SMTP:[EMAIL PROTECTED]]
> Antwoord naar: Lance Lavandowska
> Verzonden: Monday, October 11, 1999 3:43 PM
> Aan: [EMAIL PROTECTED]
> Onderwerp: Re: Real People or Search Engines?
>
> Search engines often/usually/sometimes (cover my bases) have a different
> UserAgent header than the standard browsers. You could set up a
> "pass-through" servlet to log the various UserAgents that are hitting your
> site, and continue from there.
>
> Lance Lavandowska
>
> -----Original Message-----
> From: Boemio, Neil (CAP, FGI) <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED] <[EMAIL PROTECTED]>
> Date: Sunday, October 10, 1999 11:30 PM
> Subject: Real People or Search Engines?
>
> >Is there anyway to tell if my servlet is being called by a real person
> >hitting my site or by search engines or robots or whatever.
> >
> >I ask because my servlet gives out an ID in a cookie if the user doesn't
> >have one already. And it seems that I am giving out tons of IDs now!
> How
> >can I give out IDs to "real people" ONLY?
> >
> >
___________________________________________________________________________
To unsubscribe, send email to [EMAIL PROTECTED] and include in the body
of the message "signoff SERVLET-INTEREST".
Archives: http://archives.java.sun.com/archives/servlet-interest.html
Resources: http://java.sun.com/products/servlet/external-resources.html
LISTSERV Help: http://www.lsoft.com/manuals/user/user.html
