You may want to head over to Webmonkey and read their article about Search
A few key points that have proven very helpful with my personal web site:
• A central "crawler index" with links to each page on my site that I want
spiders to crawl; this is the only page I need to submit to a search engine,
and since I store my content in MySQL the page is always up to date.
• A friendly URL structure - some search engines will follow a link like
"index.php?page=cars&subpage=ferrari", but I figured I would make my site
crawlable by *any* spider that might come its way, and began using Apache
mod_rewrite to translate urls internally; so when a browser requests a link
like "/cars/ferrari/index.html", Apache will rewrite the URL to
"index.php?page=cars&subpage=ferrari" internally. I'm not sure if this is
an issue with the CGI version of PHP like the other URL rewriting method
that has been mentioned.
Another handy tag that some spiders (Most notably, Googlebot) obey is the
ROBOTS meta tag, which tells the spider what to do with the current page:
<meta robots='index, follow'> - Index the current page, and follow all of
<meta robots='index, nofollow'> - Index the the current page, but don't
follow any links
<meta robots='noindex, follow'> - Don't index the current page, but follow
all of its links
<meta robots='noindex, nofollow'> - Don't index the current page, and don't
follow any links; dead end.
> -----Original Message-----
> From: Torkil Johnsen [mailto:[EMAIL PROTECTED]]
> the web spiders and web crawlers ever follow a link like
> index.php?page=cars&subpage=ferrari and will they ever index what
> they find
> there, and, and, and, and...
> Well... does anyone understand what I'm asking here? WOuld this be a good
> way to build my site to ensure getting hits from search engines,
> or should I
> do this some other way? Anyone know?
PHP General Mailing List (http://www.php.net/)
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
To contact the list administrators, e-mail: [EMAIL PROTECTED]