The script shipped with Nutch (bin/crawl) has the automation for crawling. It invokes a bunch of java classes in a sequence and does other stuff too. If you want to go the Eclipse way, you will have to write a java class and simulate everything that the script does...basically converting that bash code to java code. Eclipse is good option for quick debugging and running on small data. For a real crawl, Eclipse is a BAD option because your crawl might suffer if the Eclipse process crashes. Plus you don't want to die down the IDE for a long running crawl.
I would recommend you to use the script and not go for Eclipse. Thanks, Tejas On Mon, Jan 27, 2014 at 12:41 AM, rk_sharma <[email protected]> wrote: > Hi. > > I am using *nutch 2.1. in eclipse* i have done crawling using following > command > urls -threads 10 -depth 3 -topN 10 . its working fine. after this i have > done solr indexing using http://127.0.0.1:8983/solr/ -all command in > eclipse run preferences. but i need to crawl complete website how can i > proceed ? i found this > < > http://lucene.472066.n3.nabble.com/Nutch-script-to-crawl-a-whole-domain-tc3999975.html > > > question on this forum regarding my problem and it suggest that i have to > create a script for this. but how this script will work on eclipse(if i use > script). otherwise please suggest a PURE JAVA option. > > > thanks > Rajni Kant > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Crawl-a-complete-website-tp4113617.html > Sent from the Nutch - User mailing list archive at Nabble.com. >
