These test files aren't always in the same location (I never said
consistency was my strong point!) and anyway, users can always override
entries in the central analog.cfg by specifying their own .cfg file.
(I haven't tried to put that FILEEXCLUDE in the mandatory.cfg file,
though).
The reason I gave for not spidering log reports may not be a very good
one, though. Old Analog reports are often left on line, for a variety of
reasons, and so they provide links to documents that no longer exist. I
end up with 404 errors that aren't due to dead links in actual content
pages. Our search engine can also show us what page links to any given
page, but that information is a bit less meaningful if it includes the
12 archived Monthly log file reports.
I'd prefer to not index Analog reports. I can deal with the problem
fairly well by specifying my Analog report directories in the ROBOTS.TXT
file. But it would be handy to have Analog generate the exclusion
commands for me.
Aengus
______________________________ Reply Separator _________________________________
Subject: Re: [analog-help] ROBOTS Meta tag
Author: [EMAIL PROTECTED] at Internet
Date: 3/26/99 3:57 PM
Why can't you just add a
FILEEXCLUDE /test/*
to your analog.cfg file as explained in docs/include.html
Regards,
Roger Brown
On Fri, 26 Mar 1999, Aengus Lawlor wrote:
> I'm forever creating "test" web pages that aren't linked to, so that our
> Intranet spider doesn't find them and index them. (Security through
> obscurity). But recently I found that some of them were in our search
> engine, and it turned out that an Analog report had been posted and
> indexed, and the spider followed the links in the request report to find
> my "test" pages.
>
> While I immediately added the directory that the Analog report was in to
> the servers ROBOTS.TXT file, it would be handy if Analog could
> automatically add the Robot Exclusion meta tag to its output. (Now that
> I've made anlgform.exe available on the server, I can't always keep
> track where an Analog report will be posted. (And I don't have the tools
> to modify and recompile the code with the addition).
>
> The Robots exclusion tag looks like this:
>
> <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
>
> It's explained in greater detail at:
> http://info.webcrawler.com/mak/projects/robots/exclusion.html#meta
>
> Aengus
>
> --------------------------------------------------------------------
> This is the analog-help mailing list. To unsubscribe from this
> mailing list, send mail to [EMAIL PROTECTED]
> with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
> --------------------------------------------------------------------
>
--------------------------------------------------------------------
This is the analog-help mailing list. To unsubscribe from this
mailing list, send mail to [EMAIL PROTECTED]
with "unsubscribe analog-help" in the main BODY OF THE MESSAGE.
--------------------------------------------------------------------
Received: from nmho05u.rohmhaas.com ([136.141.252.23]) by ima1.rohmhaas.com with
SMTP
(IMA Internet Exchange 3.11) id 0017F4B4; Fri, 26 Mar 1999 18:06:38 -0500
Received: by nmho05u.rohmhaas.com; id SAA04755; Fri, 26 Mar 1999 18:08:32 -0500
(EST)
Received: from mb3.mailbank.com(209.133.104.8) by nmho05u.rohmhaas.com via smap
(3.2)
id xma004749; Fri, 26 Mar 99 18:08:15 -0500
Received: from gateway1.isite.net (gateway1.isite.net [198.207.204.66])
by mb3.mailbank.com (8.9.1a/8.9.1) with ESMTP id PAA08100
for <[EMAIL PROTECTED]>; Fri, 26 Mar 1999 15:06:14 -0800
Received: from proxy1.noc.isite.net ([172.16.1.11] (may be forged)) by
gateway1.isite.net (8.8.6/8.8) with ESMTP id OAA25563; Fri, 26 Mar 1999 14:58:24
-0800 (PST)
Received: from mail2.noc.isite.net (mail2.noc.isite.net [172.16.1.22]) by
proxy1.noc.isite.net (8.8.6/8.8) with ESMTP id OAA04189; Fri, 26 Mar 1999
14:58:23 -0800 (PST)
Received: (from majordom@localhost) by mail2.noc.isite.net (8.8.6/8.8) id
OAA04052 for analog-help-localoutlist; Fri, 26 Mar 1999 14:58:10 -0800 (PST)
Received: from proxy1.noc.isite.net (proxy1.noc.isite.net [172.16.1.11]) by
mail2.noc.isite.net (8.8.6/8.8) with ESMTP id OAA04048 for
<[EMAIL PROTECTED]>; Fri, 26 Mar 1999 14:58:05 -0800 (PST)
Received: from gateway1.isite.net (gateway1.isite.net [198.207.204.66]) by
proxy1.noc.isite.net (8.8.6/8.8) with ESMTP id OAA04185 for
<[EMAIL PROTECTED]>; Fri, 26 Mar 1999 14:58:05 -0800 (PST)
Received: from rogb.iserver.net (rogb.iserver.net [192.41.50.237]) by
gateway1.isite.net (8.8.6/8.8) with ESMTP id OAA25559 for
<[EMAIL PROTECTED]>; Fri, 26 Mar 1999 14:58:04 -0800 (PST)
Received: from localhost (rogb@localhost) by rogb.iserver.net (8.8.5) id
PAA24695; Fri, 26 Mar 1999 15:57:54 -0700 (MST)
Date: Fri, 26 Mar 1999 15:57:53 -0700 (MST)
From: Roger L Brown <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Subject: Re: [analog-help] ROBOTS Meta tag
In-Reply-To: <>
Message-ID: <>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Sender: [EMAIL PROTECTED]
Precedence: bulk
Reply-To: [EMAIL PROTECTED]
