Greetings Markus,
Try the attached patch.
Cheers,
Lachlan
Markus wrote:
>I've compiled ht://Dig 3.2.0b5 on WinNT 4.0 with the
>latest CygWin and GCC 3.2. When I try to dig pages
>sometimes the error occur that an empty "disallow"-line
>causes the page not to be indexed. htdig reports the
>following:
>
>Parsing robots.txt file using myname = htdig
>Robots.txt line: User-agent: *
>Found 'user-agent' line: *
>Robots.txt line: Disallow:
>Found 'disallow' line:
>Pattern:
>pushed
>Rejected: forbidden by server robots.txt!
>
>
>The robots.txt contains the following:
>
>User-agent: *
>Disallow:
>
>
>Markus
>[EMAIL PROTECTED]
--
[EMAIL PROTECTED]
ht://Dig developer DownUnder (http://www.htdig.org)
diff -r ../../cvs/htdig/htdig/Server.cc ./htdig/Server.cc
339,341c339,345
< String fullpatt = "^[^:]*://[^/]*(";
< fullpatt << pattern << ')';
< _disallow.set(fullpatt, config->Boolean("case_sensitive"));
---
> // Empty "disallow" allows all, so don't make entry which matches all.
> if (!pattern.empty())
> {
> String fullpatt = "^[^:]*://[^/]*(";
> fullpatt << pattern << ')';
> _disallow.set(fullpatt, config->Boolean("case_sensitive"));
> }
