Hello We are in the process of evaluating different opensource solutions for our distributed monitoring solution.
Currently the system is developed in house, basic features are: - there are over 100K urls to monitor at a specific interval, 1 min, 5 min, 15 min - these 100K urls are mapped to 100 parsers, for checking different syntax appear in the HTML - send out alert if parser failed While it is not exactly a crawler, but are very similar in nature. We are looking at a solution that we can focus on our business logic (i.e. the parsers), rather than the moving parts of the system (e.g. how to distribute, how to queue etc). Do you think nutch would be a good candidate? Thanks.
