Hi, Please specify what you were doing? i.e. did you run the crawl tool? what was the -depth value?
or did you use the inject and than generate and fetch. please elaborate a little. G. On Thu, 2006-01-12 at 16:44 +0800, Chih How Bong wrote: > Hi all, > I tried to invoke a indexing on 4 websites (daily news and > articles), what I got are just a scanty of web pages being indexed > (compared to if I run crawl, the pages I could index is 10 folds). I > dont know what have I don wrong or should I need to configure besides > nutch-site.xml (which I copied from nutch-default.xml). I am puzzled > thou I have read all the available tutorials. > By the way, I also noticed something strange where the crawler tried > to fetch robot.txt from each of the websites. Anyway I can disable > them, thou I have eliminated all the agents-related parameter in > nutch-site.xml. > > Thanks in advance. > > . > . > . > 161658 http.proxy.host = null > 060112 161658 http.proxy.port = 8080 > 060112 161658 http.timeout = 1000000 > 060112 161658 http.content.limit = 65536 > 060112 161658 http.agent = NutchCVS/0.7.1 (Nutch; > http://lucene.apache.org/nutch/bot.html; > [email protected]) > 060112 161658 fetcher.server.delay = 5000 > 060112 161658 http.max.delays = 10 > 060112 161659 fetching http://www.bernama.com.my/robots.txt > 060112 161659 fetching http://www.thestar.com.my/robots.txt > 060112 161659 fetching http://www.unimas.my/robots.txt > 060112 161659 fetching http://www.nst.com.my/robots.txt > 060112 161659 fetched 208 bytes from http://www.unimas.my/robots.txt > 060112 161659 fetching http://www.unimas.my/ > 060112 161659 fetched 14887 bytes from http://www.unimas.my/ > 060112 161659 fetched 204 bytes from http://www.bernama.com.my/robots.txt > 060112 161659 fetching http://www.bernama.com.my/ > 060112 161659 uncompressing.... > 060112 161659 fetched 3438 bytes of compressed content (expanded to > 10620 bytes) from http://www.nst.com.my/robots.txt > 060112 161659 fetching http://www.nst.com.my/ > 060112 161659 fetched 1181 bytes from http://www.bernama.com.my/ > 060112 161700 Using URL normalizer: org.apache.nutch.net.BasicUrlNormalizer > 060112 161701 uncompressing.... > 060112 161701 fetched 11183 bytes of compressed content (expanded to > 43846 bytes) from http://www.nst.com.my/ > 060112 161703 fetched 1635 bytes from http://www.thestar.com.my/robots.txt > 060112 161703 fetching http://www.thestar.com.my/ > 060112 161706 fetched 26712 bytes from http://www.thestar.com.my/ > 060112 161707 status: segment 20060112161614, 4 pages, 0 errors, 86626 > bytes, 9198 ms > 060112 161707 status: 0.43487716 pages/s, 73.57748 kb/s, 21656.5 bytes/page > > Rgds > Bong Chih How >
