This does lead to the question "what importance level do those <ahem>reference</ahem> links actually have to offer a spider?" Is the content supplier making a default statement on the value of the references therein to a spider? Such as: The wapper "reference" ads links that I will not show you Mr spider have zero or near zero value in relation to the centre content of the page and those links therein that body of information. Careful, those links may be the IE user eye ball grabbing links that say "click me" for something completely different Mr Monty Pythn. Take care that your link reference counts that apply referencing "value or weight" are not skewed. It may be wise to compare the links returned with and then without the IE agent identity being assumed by the spider. Try to apply that comparative knowledge in link weight evaluations for the referencial strength assessment you do on the content of the page. (until the Semantic web does this ) If the body links are ok, I generally have a higher trust level on those links being real reference links. Content creators have a higher "due diligence" for their article then the content publishers that add content wrapper style navigation, IMHO; but I think everyone can give you examples of the other extreme. ( yet would not those excellent publishers take steps to allow your spider a quality experience in walking their offerings without the need for an IE agent disguise??? so as to being in more readers to a quality site???) -Thomas Kay ---------- Original Text ---------- From: "Erick Thompson" <[EMAIL PROTECTED]>, on 28/02/2002 2:32 PM: A lot of navigation scripts are setup to work with IE or Netscape, and the fall through navigation html is sometimes incomplete or doesn't work at all. Erick ----- Original Message ----- From: "Oskar Bartenstein" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Wednesday, February 27, 2002 9:06 PM Subject: [Robots] Re: User agent > > What would be a good motivation to do so? > Server side benefit? Spider side benefit? > > Wed, 27 Feb 2002 19:23:01 -0800 "Erick Thompson" <[EMAIL PROTECTED]> said: > > I'm trying to constructor a user agent string that emulates IE5/6, but also > > -- > Dr. Oskar Bartenstein [EMAIL PROTECTED] > IF Computer Japan http://www.ifcomputer.com > > > -- > This message was sent by the Internet robots and spiders discussion list ([EMAIL PROTECTED]). For list server commands, send "help" in the body of a message to "[EMAIL PROTECTED]". -- This message was sent by the Internet robots and spiders discussion list ([EMAIL PROTECTED]). For list server commands, send "help" in the body of a message to "[EMAIL PROTECTED]". -- This message was sent by the Internet robots and spiders discussion list ([EMAIL PROTECTED]). For list server commands, send "help" in the body of a message to "[EMAIL PROTECTED]".
