Droids Features

[Grant Ingersoll]

Is there a feature list for Droids anywhere?

Or, can it do:
1. Honor robots.txt
2. Crawl throttling
3. Distributed crawling (i.e. give a bunch of links to it and some  
distributed compute resources and have it go to town)

Thanks,
Grant

---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]