Does HttpClient have anything to parse a robots.txt file? If not, would anyone be interested in http://www.osjava.org/norbert/ ?
I'd like to put it in the sandbox and thought that it would be of a lot of interest to the HttpClient project and users. It would need adjusting to sit on top of HttpClient as it currently uses the JDK to download the robots.txt file itself, but that shouldn't be very hard. Equally, HttpClient might want to, by default, refuse to download things if it's against the robots.txt rules and make people configure HttpClient to ignore the robots.txt to get around it. Hen --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
