Dear all. I have a problem with nutch Internet crawl/recrawl script (I'm wanted to understand how it works, so I wrote it by myself).
After I merge indexes (merging segments seems to be fine), I search doesn't work for me: $ bin/nutch org.apache.nutch.searcher.NutchBean http Total hits: 0 Before recrawling I was able to search (index was placed at crawl/indexes) My script: --------------------------------------------- #!/bin/bash export JAVA_HOME=/usr/lib/jvm/java-6-sun #Inject new urls bin/nutch inject crawl/crawldb dmoz/urls echo "new URLs injected (dmoz/urls)" #generate segments bin/nutch generate crawl/crawldb crawl/segments -topN $3 echo "segments generated" #generate fetch-list s1=`ls -d crawl/segments/2* | tail -1` echo $s1 echo "fetch-list generated" #fetch bin/nutch fetch $s1 -threads $2 echo "fetching done" #update the database with results of fetch bin/nutch updatedb crawl/crawldb $s1 echo "database updated" #merge segments bin/nutch mergesegs crawl/MERGEDsegments crawl/segments/* rm -r crawl/segments mv crawl/MERGEDsegments crawl/segments echo "segments merged" #inverting links bin/nutch invertlinks crawl/linkdb -dir crawl/segments echo "links inverted" #indexing bin/nutch index crawl/NEWindexes crawl/crawldb crawl/linkdb crawl/segments/* echo "indexing done" #dedup - delete duplicate documents in the index bin/nutch dedup crawl/NEWindexes echo "dedup done" #merging indexes bin/nutch merge crawl/MERGEDindexes crawl/NEWindexes echo "indexes merged" # replace indexes with indexes_merged mv --verbose crawl/indexes crawl/OLDindexes mv --verbose crawl/MERGEDindexes crawl/indexes/part-00000 #clean up rm -rf crawl/NEWindexes rm -rf crawl/OLDindexes ------------------------------------------------- What's wrong with the script? Thank You in advance, Kind Regards, -- Andrey Sapegin, Software Developer, Unister GmbH [email protected] www.unister.de
