Blake Winton <[EMAIL PROTECTED]> wrote: > wondered if Plucker wanted to start being a good web-citizen > and honouring robots.txt.
Plucker is not necessarily a robot, although it can operate as one. It should support robots.txt only when it is recursing into a site, not when downloading a single page, IMO. It should always support the robots meta. If that is acceptable and no-one beats me to it, I will try to code this into the main plucker-build parser later this week and send a patch. _______________________________________________ plucker-dev mailing list [EMAIL PROTECTED] http://lists.rubberchicken.org/mailman/listinfo/plucker-dev
