On Mon, 29 Jan 2001, Dennis Nichols wrote:
Greetings - First, my understanding of ROBOTINCLUDE and ROBOTEXCLUDE is
that these are report-level commands, that is, they affect only the
Operating System Report. Right?
Right. At the moment.
A previous exchange on this list...
On Thu, 25
On 30 Jan 2001, at 11:53, Stephen Turner wrote about
Re: [analog-help] robots, including, excluding, a:
ROBOTINCLUDE -FILE filename
ROBOTEXCLUDE -FILE filename
BROWEXCLUDE -FILE filename
BROWINCLUDE -FILE filename
This differs from CONFIGFILE because only the arguments would
In [EMAIL PROTECTED], Stephen
Turner [EMAIL PROTECTED] writes:
Well, I guess the point is that Aaron's suggestion would break the idea of
ROBOT*CLUDE being a report-level command. So then it wouldn't really be
contradictory.
At the risk of breaking backward compatibility I would prefer to
Stephen Turner wrote:
How about this instead/in addition:
For a selected set of commands, invent a syntax extension that says read
the arguments for this command from a file. I could then, for example, put
a list of robotish browsers in a file and use any of the following:
At 1/30/01 01:56 PM, Klaus Johannes Rusch wrote:
In [EMAIL PROTECTED],
Stephen Turner [EMAIL PROTECTED] writes:
ROBOTINCLUDE -FILE filename
ROBOTEXCLUDE -FILE filename
BROWEXCLUDE -FILE filename
BROWINCLUDE -FILE filename
Yes, it makes sense. I'm not sure whether I like it as an
Greetings - First, my understanding of ROBOTINCLUDE and ROBOTEXCLUDE is
that these are report-level commands, that is, they affect only the
Operating System Report. Right?
A previous exchange on this list...
On Thu, 25 Jan 2001, Stephen Turner wrote:
On Wed, 24 Jan 2001, Aaron Shoblaske
On Thu, 2 Mar 2000, Alejandro Fernandez wrote:
I get the following warning though:
/analog/analog: Warning M: Logfile - contains lines with no browsers,
which are being filtered
Does this mean that if a logfie entry is without a browser entry , it doesn't
show up in the stats?
No,
Hi,
My company's being audited this week, and the people checking up on us have a
long list of robots to exclude, some of which have spaces in their name.
A quick browse through the analog docs says
BROWEXCLUDE "lycos spider"
could be the best way of excluding them, but it doesn't work. They
On Thu, 2 Mar 2000, Alejandro Fernandez wrote:
Hi,
My company's being audited this week, and the people checking up on us have a
long list of robots to exclude, some of which have spaces in their name.
A quick browse through the analog docs says
BROWEXCLUDE "lycos spider"
could be
Lycos' spider's agent name is Lycos_Spider_(T-Rex)
and its host name is: *.sjc.lycos.com (e.g. sjc-fe4-1.sjc.lycos.com)
A great table that lists most of the majors is available at:
http://searchenginewatch.com/webmasters/spiderchart.html
If you have a robot.txt file on your site you could
On Thu, 02 Mar 2000, you wrote:
On Thu, 2 Mar 2000, Alejandro Fernandez wrote:
Hi,
My company's being audited this week, and the people checking up on us have a
long list of robots to exclude, some of which have spaces in their name.
A quick browse through the analog docs says
We use anonymous cookies to identify unique visitors to our sites and
determine things like requests/visitor, etc. It's not perfect but it works
pretty well, as long as your time-scale is short enough that you can make
insignificant the probability that many users have lost/deleted their
Matt,
You could try looking at the referrer info -- are the "shoppers" connecting
from some site that is sending you traffic or is it just that they are "typing
in" your URL. Robots are not likely to carry referrer data.
Second, you could do some IP/DNS reasearch. Where are the "shoppers"
ever heard of), but
is out of date for some of those entries that I've checked.
Aengus
__ Reply Separator _
Subject: Re: [analog-help] ROBOTS Meta tag
Author: [EMAIL PROTECTED] at Internet
Date:3/27/99 7:42 PM
On Fri, 26 Mar 1999, Aengus La
On Fri, 26 Mar 1999, Aengus Lawlor wrote:
The Robots exclusion tag looks like this:
META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"
It's explained in greater detail at:
http://info.webcrawler.com/mak/projects/robots/exclusion.html#meta
It's a nice idea, but does anyone know which
I'm forever creating "test" web pages that aren't linked to, so that our
Intranet spider doesn't find them and index them. (Security through
obscurity). But recently I found that some of them were in our search
engine, and it turned out that an Analog report had been posted and
indexed, and
16 matches
Mail list logo
