On Thursday, August 7, 2003, at 08:57 AM, Scott Cadillac wrote:
Roland and I were corresponding last night off-list.
After I apologized to him of course, for taking over the thread that he started :-)
Happy to have wiser folks take the lead, particularly in something that is confusing and complex.
One simple solution is to not let the search bots into the site or not into areas that use <@USERREFERENCEARGUMENT>
I do notice that most are polite enough to check for robots.txt.
Among some other ideas, I suggested the following:
----------
Using the <@CGIPARAM USER_AGENT> Metatag, you could identify when a
SearchBot has just hit your site. I've noticed in my server logs that most
every SearchBot has some unique UserAgent string that is not impossible to
identify.
Once you've identified a SearchBot, then you could assign a special variable
that then hides all your <@USERREFERENCEARGUMENT> Metatags, like so:
<a href="<@APPFILE>?<@IFEMPTY <@VAR local$ImABot>><@USERREFERENCEARGUMENT></@IF>">A link</a>
Then the _UserReference value never gets captured to begin with.
----------
At the top of your TAFs, like in a common TCF, you would need some logic
that parses for Bots.
Here are some bot agents I got from last night's log:
FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp) Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org) Mozilla/2.0 (compatible; Ask Jeeves/Teoma) Mozilla/4.0 compatible ZyBorg/1.0 DLC ([EMAIL PROTECTED]; http://www.WISEnutbot.com) Zeus 2.6 Googlebot/2.1 (+http://www.googlebot.com/bot.html)
From the above we can get some keywords like:
FAST-WebCrawler grub-client Ask Jeeves/Teoma www.WISEnutbot.com Zeus 2.6 Googlebot/2.1
Then we stored the keywords in an Array (Domain or Custom Scope), then we
could loop on these keyword like:
<@ROWS ARRAY=BotScope$Bots>
<@IF EXPR="<@CGIPARAM USER_AGENT> contains '<@VAR BotScope$Bots[1,1]>'">
<@ASSIGN local$ImABot VALUE="1">
</@IF
</@ROWS>
The trick is that you couldn't store this Variable in User Scope - because
remember you are not supplying <@UserReferenceArgument> and Bot clients
don't typically accept cookies (session or otherwise).
This new scrap of code could also easily be wrapped in a Custom Metatag, and
then we have <@URA2> or something nice and short like that.
Yes, it seems like a lot of work at first, but a global search-and-replace
with a tool like http://www.vknoware.com/afr/index.htm makes things a lot
easier.
Just an idea. Now I'll say no more on the subject :-)
Cheers.....
Scott Cadillac, Witango.org - http://witango.org 403-281-6090 - [EMAIL PROTECTED] -- Information for the Witango Developer Community ---------------------
XML-Extranet - http://xml-extra.net 403-281-6090 - [EMAIL PROTECTED] -- Well-formed Development (for hire) ---------------------
_______________________________________________________________________ _
TO UNSUBSCRIBE: Go to http://www.witango.com/maillist.taf
________________________________________________________________________ TO UNSUBSCRIBE: Go to http://www.witango.com/maillist.taf
