Suppose I have a site. A fairly large and popular site that is some sort of message board type site, with several million or so unique and legitimate messages.
Suppose the URLs for the messages are all in the form http://site/foo/showme.foo?msgid=6666 where 6666 identififes the message. Suppose I want common robots to index it, since the messages do contain useful content, so it is to their advantage because it gives them real and useful content that gives them better results than other engines that don't index them, and to my advantage to have it indexed since it brings people to the site. Am I out of luck with most of them because most robots won't spider URLs with "?"s in them? Any choice but to convert to using a URL format that doesn't include a "?"? That is doable, but is obviously a big pain. Does anyone know if popular search engine companies can be convinced to include such URLs if specially requested for a particular site (and if I can assure them that they won't end up wandering an infinite virtual URL space), and how that convincing would be done? Thanks.
