I am using a script similar to that of the 0.8 Recrawl script on the Wiki. For my crawl, I am specifying a depth of 20. My problem is that often times when I crawl, there are no more pages to crawl once it reaches a depth of level 15, but the generate/fetch/update cycle continues to execute a total of 20 times, generating/'fetching' 5 segments that arent needed/are-empty.
I would like to programmatically detect that there are no more pages to crawl, and then terminate the crawl. So in other words, I need to figure out how to tell that my last call to Generate has created a segment with no pages to fetch. Can any one offer any suggestions towards this goal? Thanks! Jared- ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
