> > From what I have been able to find, the verity spider will
> > only spider sites on a single network domain... so a remote
> > machine must reside on the same domain. What I'm wondering
> > is if anybody
>
> Yup. Verity (and MS Index Server) are what's called "Worms" -
> they "burrow" t
ilto:[EMAIL PROTECTED]]
> Sent: February 5, 2003 9:45 AM
> To: CF-Talk
> Subject: RE: Remotely spider websites
>
>
> > -Original Message-
> > From: Jillian Carroll [mailto:[EMAIL PROTECTED]]
> > Sent: Wednesday, February 05, 2003 10:18 AM
> > To: CF-Talk
&g
Let me add:
Please, for the love of Pete, respect the directives in the robots.txt file.
> -Original Message-
> From: Jim Davis [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, February 05, 2003 10:45 AM
> To: CF-Talk
> Subject: RE: Remotely spider websites
>
>
>
> -Original Message-
> From: Jillian Carroll [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, February 05, 2003 10:18 AM
> To: CF-Talk
> Subject: Remotely spider websites
>
>
> >From what I have been able to find, the verity spider will
> only spider
>
>From what I have been able to find, the verity spider will only spider
sites on a single network domain... so a remote machine must reside on
the same domain. What I'm wondering is if anybody can recommend (or has
written) a tool that will let me spider remote sites in a CF-friendly
manner?
Tha
5 matches
Mail list logo