On Wed, 26 Jan 2005, Nick Rout wrote:
Mozilla/5.0 (compatible; Googlebot/2.1+http://www.google.com/bot.html)
Ooo. I _like_ that! So all those snooty sites that hide their content behind "You need IE to view this site" are not getting googled. Yet another reason to love Google!
I'm not so sure that such silliness prevents spidering
Oh but it does, the class of site I'm talking about checks the UserAgent string and refuses to display it content unless it is being browsed by IE.
So if Google is reporting Mozilla as its useragent, then this class of broken site will simply give Google it's content/internal link free "install IE" page.
There is no way for Google to spider past it!
Instant web site suicide.
Think of it as evolution in action.
John Carter Phone : (64)(3) 358 6639 Tait Electronics Fax : (64)(3) 359 4632 PO Box 1645 Christchurch Email : [EMAIL PROTECTED] New Zealand
"The notes I handle no better than many pianists. But the pauses between the notes - ah, that is where the art resides!' - Artur Schnabel
