On Fri, 2008-09-12 at 10:12 -0400, Grant Ingersoll wrote: > Additionally, how hard would it be to add Crawlers for things like:
Droids does not force you to use anything besides the queue. From there you can do what you want. > > 1. IMAP and other mail stores (even things like PST files, etc.) One can add new protocol implementation in no time. > 2. Somewhat strange: Databases. Just point it at a DB and have it > suck in tables/rows/columns I did this in one of my droids @work. Actually it was part of the business logic and my droid only invokes this business logic. > 3. Things like web APIs (Flickr, del.icio.us, etc), That should be as well very easy. To add basic support for them would be a couple of hours and an advance support depends on the API. > > Any comments on fault tolerance and incremental crawling would also be > appreciated. That would need some small enhancements in the queue (basically adding a comparison on the last modified date). Should not be a problem. > Is there anything in the current design that you think > prevents these things? No. See the great example that Oleg wrapped up in no time. He implemented a simpleRuntime without using Spring. Droids follows the slogan "Impossible is nothing". The great flexibility of droids is/should/will be the great feature of Droids. salu2 > > Thanks, > Grant > > On Aug 27, 2008, at 5:26 PM, Grant Ingersoll wrote: > > > Is there a feature list for Droids anywhere? > > > > Or, can it do: > > > > 1. Honor robots.txt > > 2. Crawl throttling > > 3. Distributed crawling (i.e. give a bunch of links to it and some > > distributed compute resources and have it go to town) > > > > Thanks, > > Grant > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > -- Thorsten Scherler thorsten.at.apache.org Open Source Java consulting, training and solutions --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
