Howard Ha wrote:
>
> Hi there,
>
> I am experiencing a weird problem and I'm not sure what the cause might
> be. If I am digging a relatively small site everything works fine. I can
> dig pretty large numbers of pages without problems, but when I try to add
> more sites to dig I get this problem where everything just stops.
Have you tried digging them one at a time? I suspect that you're
probably running into a robots.txt problem.
Good luck,
Doug
------------------------------------
To unsubscribe from the htdig mailing list, send a message to
[EMAIL PROTECTED] containing the single word unsubscribe in
the SUBJECT of the message.
