Hi, it seems to me that RobotRules.pm and RobotUA.pm don't quite agree on what makes up the $netloc handle used to access the robot rules. simple_request() in RobotUA.pm takes the URI::host_port() function for host_wait() and visit() calls, while allowed() in RobotRules uses URI::authorize() wherever it has to extract a netloc from an URI. Unfortunately, authorize() and host_port() themselfs do not agree whether a port should be given when it is the default port for this scheme. -- Thomas Richter
