Doh! Found the jar ;-) Out of interest are there any nutch maven2 repositories out there?
Cheers Aled -----Original Message----- From: Aled Rhys Jones [mailto:[EMAIL PROTECTED] Sent: 28 December 2007 13:19 To: '[email protected]' Subject: RE: A few nutch questions Thanks Jeff! You say you created a NutchBean? I assume you mean you created a bean in your application that makes use of the NutchBean? Which Jar's do I need in my application to be able to execute search on NutchBean? It looks like I need to pass in a Hadoop Configuration object that is populated with site resource data. Thanks again Aled -----Original Message----- From: Bolle, Jeffrey F. [mailto:[EMAIL PROTECTED] Sent: 27 December 2007 19:16 To: [email protected] Subject: RE: A few nutch questions Aled, I've integrated Nutch into my current application by simply creating a NutchBean and pointing it at the configuration files and indexes. It works great for me. http://lucene.apache.org/nutch/apidocs/org/apache/nutch/searcher/NutchB ean.html That is the JavaDoc API page for the Nutch Bean. It is pretty self explanatory and it took me about 2 hours to get things working the way I wanted them to. Good luck. Jeff -----Original Message----- From: Aled Rhys Jones [mailto:[EMAIL PROTECTED] Sent: Thursday, December 27, 2007 11:12 AM To: [email protected] Subject: A few nutch questions Hi everyone I've got a few questions on Nutch, we're using the latest release version - 0.9. We're looking to crawl, index and search roughly 20,000 websites. Based on the example of roughly 10KB per page, and a conservative estimate of 100 pages per site (since the sites will be commercial in nature, my guess is that most will have more than that), then it equates to about 20GB of storage required. Does anyone have some stats on roughly how much bandwidth is required to crawl this amount of sites once? We've tried to start out small, but already our dedicated server host is complaining about outbound bandwidth and breaking terms of use. Are there any recommended hosts for crawling using Nutch, or can anyone recommend hosting particulars to look out for? Lastly, we'd like to connect to our nutch install via an API, so we can add more content to our results. Our main application is a standard java web application with spring, hibernate, mysql etc sitting pretty on tomcat. We also (currently) run Nutch on the same tomcat install. What's the best way to communicate to the nutch install from our current application to provide it with search engine capabilities? Does nutch include web services, or can we use RMI? Thank you for any feedback, it would be greatly appreciated. Cheers Aled
