[analog-help] Which URLs do the search engines index?

[Michael D. Crawford]

This isn't related to Analog, exactly, but I use Analog to monitor my 
site's traffic, and it seems to me that some of the folks here should 
know the answer to my question, and most would care about the answer too.

I'm swapping links with lots of other sites to try to improve the search 
engine rank of some of my pages.

But I'm concerned that some of the pages that other people place my link 
on are not accessible to the search engines, so they would then not 
contribute to improved positioning.

One problem is that the sites don't always provide a path to their 
reciprocal links page.  But I can work around that by submitting the URL 
to the new URL form at some of the search engines.  For the most part I 
only ever submit it to Google at http://www.google.com/addurl.html

But search engines don't generally index all the pages a site has.  I 
recall reading on the Enhydra mailing list (http://www.enhydra.org/) 
that search engines won't index Enhydra's ".po" files, so some monkeying 
with URL rewriting is necessary to make the files appear to be ".html" 
files when they appear on the web.

I had been under the impression that URLs with parameters in them don't 
get indexed, that is, a page where "&" or "?" appears in the title.  But 
it appears (happily) that I am wrong.  This page:

http://www.edinburgh-hotels.ws/other_resources.asp?l=1&group=27&groupname=Other
uses parameters, but I find that it shows up in Google's cache when I 
search for that page.  Happily so, as they've linked my site.

Could it be that such pages are always indexed, if the site operator 
doesn't use a robots.txt?

Generally, what are the rules commonly used by search engines?  What 
file suffixes get index, and which don't?  Was I completely mistaken 
about them not indexing URLs with parameters?

Thanks for your help!
Michael D. Crawford
[EMAIL PROTECTED]
      Read "GoingWare's Bag of Programming Tricks" at:
               http://www.goingware.com/tips/
+------------------------------------------------------------------------
|  TO UNSUBSCRIBE from this list:
|    http://lists.meer.net/mailman/listinfo/analog-help
|
|  Usenet version: news://news.gmane.org/gmane.comp.web.analog.general
|  List archives:  http://www.analog.cx/docs/mailing.html#listarchives
+------------------------------------------------------------------------