Author: Ivan Mikhnevich
Email: [EMAIL PROTECTED]
Message:
Hello, everybody and especially developers of mnoGoSearch.
I've found a bug in robots.txt standard implementation. The only path, without
filenames, is compared with Disallow-lines from robots.txt. While the standard at
http://www.robotstxt.org/wc/norobots.html declares that Disallow line should be
compared with path with filename.
The following code fixes it:
mnoGoSeach 3.1.12, robots.c (line 29)
UDM_ROBOT * UdmFindRobots(UDM_ENV * Conf,UDM_URL * URL){
size_t i;
char path[UDM_URLSIZE];
strcpy(path, URL->path);
strcat(path, URL->filename);
for(i=0;i<Conf->Robots.nrobots;i++){
if(!strcmp(URL->hostinfo,Conf->Robots.robots[i].hostinfo)&&!strncmp(path,Conf->Robots.robots[i].path,strlen(Conf->Robots.robots[i].path)))
return(&Conf->Robots.robots[i]);
}
return(NULL);
}
Regards, Ivan
Reply: <http://search.mnogo.ru/board/message.php?id=2120>
___________________________________________
If you want to unsubscribe send "unsubscribe general"
to [EMAIL PROTECTED]
