Hey guys, Alright, so I've been able to succesfully crawl a wiki on the domain name wwaps.blah.domain.com/.
However, I configure it to crawl all intranet web pages at intranet.corp.domain.com and start the crawl. However, it only seems to fetch 6 pages or so then says that the indexing is completed. Any ideas? Thanks! Matt. The output of the crawl is below: 060608 112059 parsing file:/home/rdu/mholt/Desktop/nutch-0.7.2/conf/nutch-default.xml 060608 112059 parsing file:/home/mholt/Desktop/nutch-0.7.2/conf/crawl-tool.xml 060608 112059 parsing file:/home/mholt/Desktop/nutch-0.7.2/conf/nutch-site.xml 060608 112059 No FS indicated, using default:local 060608 112059 crawl started in: crawl/ 060608 112059 rootUrlFile = urls/redhat 060608 112059 threads = 10 060608 112059 depth = 10 060608 112059 Created webdb at LocalFS,/home/mholt/Desktop/nutch-0.7.2/crawl/db 060608 112059 Starting URL processing 060608 112059 Plugins: looking in: /home/mholt/Desktop/nutch-0.7.2/plugins 060608 112059 not including: /home/mholt/Desktop/nutch-0.7.2/plugins/clustering-carrot2 060608 112059 not including: /home/mholt/Desktop/nutch-0.7.2/plugins/creativecommons 060608 112059 parsing: /home/mholt/Desktop/nutch-0.7.2/plugins/index-basic/plugin.xml 060608 112059 impl: point=org.apache.nutch.indexer.IndexingFilter class=org.apache.nutch.indexer.basic.BasicIndexingFilter 060608 112059 not including: /home/mholt/Desktop/nutch-0.7.2/plugins/index-more 060608 112059 not including: /home/mholt/Desktop/nutch-0.7.2/plugins/language-identifier 060608 112059 parsing: /home/mholt/Desktop/nutch-0.7.2/plugins/nutch-extensionpoints/plugin.xml 060608 112059 not including: /home/mholt/Desktop/nutch-0.7.2/plugins/ontology 060608 112059 not including: /home/mholt/Desktop/nutch-0.7.2/plugins/parse-ext 060608 112059 parsing: /home/mholt/Desktop/nutch-0.7.2/plugins/parse-html/plugin.xml 060608 112059 impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.html.HtmlParser 060608 112059 not including: /home/mholt/Desktop/nutch-0.7.2/plugins/parse-js 060608 112059 not including: /home/mholt/Desktop/nutch-0.7.2/plugins/parse-msword 060608 112059 not including: /home/mholt/Desktop/nutch-0.7.2/plugins/parse-pdf 060608 112059 not including: /home/mholt/Desktop/nutch-0.7.2/plugins/parse-rss 060608 112059 parsing: /home/mholt/Desktop/nutch-0.7.2/plugins/parse-text/plugin.xml 060608 112059 impl: point=org.apache.nutch.parse.Parser class=org.apache.nutch.parse.text.TextParser 060608 112059 not including: /home/mholt/Desktop/nutch-0.7.2/plugins/protocol-file 060608 112059 not including: /home/mholt/Desktop/nutch-0.7.2/plugins/protocol-ftp 060608 112059 parsing: /home/mholt/Desktop/nutch-0.7.2/plugins/protocol-http/plugin.xml 060608 112059 impl: point=org.apache.nutch.protocol.Protocol class=org.apache.nutch.protocol.http.Http 060608 112059 not including: /home/mholt/Desktop/nutch-0.7.2/plugins/protocol-httpclient 060608 112059 parsing: /home/mholt/Desktop/nutch-0.7.2/plugins/query-basic/plugin.xml 060608 112059 impl: point=org.apache.nutch.searcher.QueryFilter class=org.apache.nutch.searcher.basic.BasicQueryFilter 060608 112059 not including: /home/mholt/Desktop/nutch-0.7.2/plugins/query-more 060608 112059 parsing: /home/mholt/Desktop/nutch-0.7.2/plugins/query-site/plugin.xml 060608 112059 impl: point=org.apache.nutch.searcher.QueryFilter class=org.apache.nutch.searcher.site.SiteQueryFilter 060608 112059 parsing: /home/mholt/Desktop/nutch-0.7.2/plugins/query-url/plugin.xml 060608 112059 impl: point=org.apache.nutch.searcher.QueryFilter class=org.apache.nutch.searcher.url.URLQueryFilter 060608 112059 not including: /home/mholt/Desktop/nutch-0.7.2/plugins/urlfilter-prefix 060608 112059 parsing: /home/mholt/Desktop/nutch-0.7.2/plugins/urlfilter-regex/plugin.xml 060608 112059 impl: point=org.apache.nutch.net.URLFilter class=org.apache.nutch.net.RegexURLFilter 060608 112059 found resource crawl-urlfilter.txt at file:/home/mholt/Desktop/nutch-0.7.2/conf/crawl-urlfilter.txt 060608 112059 Using URL normalizer: org.apache.nutch.net.BasicUrlNormalizer 060608 112059 Added 1 pages 060608 112059 Processing pagesByURL: Sorted 1 instructions in 0.0060 seconds. 060608 112059 Processing pagesByURL: Sorted 166.66666666666666 instructions/second 060608 112059 Processing pagesByURL: Merged to new DB containing 1 records in 0.0010 seconds 060608 112059 Processing pagesByURL: Merged 1000.0 records/second 060608 112059 Processing pagesByMD5: Sorted 1 instructions in 0.0050 seconds. 060608 112059 Processing pagesByMD5: Sorted 200.0 instructions/second 060608 112059 Processing pagesByMD5: Merged to new DB containing 1 records in 0.0 seconds 060608 112059 Processing pagesByMD5: Merged Infinity records/second 060608 112059 Processing linksByMD5: Copied file (4096 bytes) in 0.0090 secs. 060608 112059 Processing linksByURL: Copied file (4096 bytes) in 0.0080 secs. 060608 112059 FetchListTool started 060608 112100 Processing pagesByURL: Sorted 1 instructions in 0.0050 seconds. 060608 112100 Processing pagesByURL: Sorted 200.0 instructions/second 060608 112100 Processing pagesByURL: Merged to new DB containing 1 records in 0.0010 seconds 060608 112100 Processing pagesByURL: Merged 1000.0 records/second 060608 112100 Processing pagesByMD5: Sorted 1 instructions in 0.0040 seconds. 060608 112100 Processing pagesByMD5: Sorted 250.0 instructions/second 060608 112100 Processing pagesByMD5: Merged to new DB containing 1 records in 0.0010 seconds 060608 112100 Processing pagesByMD5: Merged 1000.0 records/second 060608 112100 Processing linksByMD5: Copied file (4096 bytes) in 0.0070 secs. 060608 112100 Processing linksByURL: Copied file (4096 bytes) in 0.014 secs. 060608 112100 Processing /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112100/fetchlist.unsorted: Sorted 1 entries in 0.0050 seconds. 060608 112100 Processing /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112100/fetchlist.unsorted: Sorted 200.0 entries/second 060608 112100 Overall processing: Sorted 1 entries in 0.0050 seconds. 060608 112100 Overall processing: Sorted 0.0050 entries/second 060608 112100 FetchListTool completed 060608 112100 logging at INFO 060608 112100 fetching http://intranet.corp.domain.com/ic/intranet/ 060608 112100 http.proxy.host = null 060608 112100 http.proxy.port = 8080 060608 112100 http.timeout = 10000 060608 112100 http.content.limit = 65536 060608 112100 http.agent = NutchCVS/0.7.2 (Nutch; http://lucene.apache.org/nutch/bot.html; [email protected]) 060608 112100 fetcher.server.delay = 1000 060608 112100 http.max.delays = 100 060608 112101 status: segment 20060608112100, 1 pages, 0 errors, 39453 bytes, 1013 ms 060608 112101 status: 0.9871668 pages/s, 304.27103 kb/s, 39453.0 bytes/page 060608 112102 Updating /home/mholt/Desktop/nutch-0.7.2/crawl/db 060608 112102 Updating for /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112100 060608 112102 Processing document 0 060608 112102 Finishing update 060608 112102 Processing pagesByURL: Sorted 7 instructions in 0.0060 seconds. 060608 112102 Processing pagesByURL: Sorted 1166.6666666666667 instructions/second 060608 112102 Processing pagesByURL: Merged to new DB containing 6 records in 0.0010 seconds 060608 112102 Processing pagesByURL: Merged 6000.0 records/second 060608 112102 Processing pagesByMD5: Sorted 7 instructions in 0.0040 seconds. 060608 112102 Processing pagesByMD5: Sorted 1750.0 instructions/second 060608 112102 Processing pagesByMD5: Merged to new DB containing 6 records in 0.0010 seconds 060608 112102 Processing pagesByMD5: Merged 6000.0 records/second 060608 112102 Processing linksByMD5: Sorted 7 instructions in 0.0060 seconds. 060608 112102 Processing linksByMD5: Sorted 1166.6666666666667 instructions/second 060608 112102 Processing linksByMD5: Merged to new DB containing 5 records in 0.0020 seconds 060608 112102 Processing linksByMD5: Merged 2500.0 records/second 060608 112102 Processing linksByURL: Sorted 5 instructions in 0.054 seconds. 060608 112102 Processing linksByURL: Sorted 92.5925925925926 instructions/second 060608 112102 Processing linksByURL: Merged to new DB containing 5 records in 0.0020 seconds 060608 112102 Processing linksByURL: Merged 2500.0 records/second 060608 112102 Processing linksByMD5: Sorted 5 instructions in 0.0050 seconds. 060608 112102 Processing linksByMD5: Sorted 1000.0 instructions/second 060608 112102 Processing linksByMD5: Merged to new DB containing 5 records in 0.0010 seconds 060608 112102 Processing linksByMD5: Merged 5000.0 records/second 060608 112102 Update finished 060608 112102 FetchListTool started 060608 112103 Processing pagesByURL: Sorted 5 instructions in 0.0040 seconds. 060608 112103 Processing pagesByURL: Sorted 1250.0 instructions/second 060608 112103 Processing pagesByURL: Merged to new DB containing 6 records in 0.0010 seconds 060608 112103 Processing pagesByURL: Merged 6000.0 records/second 060608 112103 Processing pagesByMD5: Sorted 5 instructions in 0.0040 seconds. 060608 112103 Processing pagesByMD5: Sorted 1250.0 instructions/second 060608 112103 Processing pagesByMD5: Merged to new DB containing 6 records in 0.0010 seconds 060608 112103 Processing pagesByMD5: Merged 6000.0 records/second 060608 112103 Processing linksByMD5: Copied file (4096 bytes) in 0.0070 secs. 060608 112103 Processing linksByURL: Copied file (4096 bytes) in 0.016 secs. 060608 112103 Processing /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112103/fetchlist.unsorted: Sorted 5 entries in 0.0040 seconds. 060608 112103 Processing /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112103/fetchlist.unsorted: Sorted 1250.0 entries/second 060608 112103 Overall processing: Sorted 5 entries in 0.0040 seconds. 060608 112103 Overall processing: Sorted 8.0E-4 entries/second 060608 112103 FetchListTool completed 060608 112103 logging at INFO 060608 112103 fetching http://intranet.corp.domain.com/ic/intranet/backlinks.html 060608 112103 fetching http://intranet.corp.domain.com/cgi-bin/search.cgi 060608 112103 fetching http://intranet.corp.domain.com/ic/intranet/login.html 060608 112103 fetching http://intranet.corp.domain.com/ic/intranet/index.html 060608 112103 fetching http://intranet.corp.domain.com/ic/intranet/IntranetHelp 060608 112115 status: segment 20060608112103, 5 pages, 0 errors, 161547 bytes, 12027 ms 060608 112115 status: 0.41573125 pages/s, 104.93771 kb/s, 32309.4 bytes/page 060608 112116 Updating /home/mholt/Desktop/nutch-0.7.2/crawl/db 060608 112116 Updating for /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112103 060608 112116 Processing document 0 060608 112116 Finishing update 060608 112116 Processing pagesByURL: Sorted 21 instructions in 0.0060 seconds. 060608 112116 Processing pagesByURL: Sorted 3500.0 instructions/second 060608 112116 Processing pagesByURL: Merged to new DB containing 6 records in 0.0010 seconds 060608 112116 Processing pagesByURL: Merged 6000.0 records/second 060608 112116 Processing pagesByMD5: Sorted 10 instructions in 0.0040 seconds. 060608 112116 Processing pagesByMD5: Sorted 2500.0 instructions/second 060608 112116 Processing pagesByMD5: Merged to new DB containing 6 records in 0.0010 seconds 060608 112116 Processing pagesByMD5: Merged 6000.0 records/second 060608 112116 Processing linksByMD5: Sorted 21 instructions in 0.0070 seconds. 060608 112116 Processing linksByMD5: Sorted 3000.0 instructions/second 060608 112116 Processing linksByMD5: Merged to new DB containing 15 records in 0.0020 seconds 060608 112116 Processing linksByMD5: Merged 7500.0 records/second 060608 112116 Processing linksByURL: Sorted 15 instructions in 0.061 seconds. 060608 112116 Processing linksByURL: Sorted 245.9016393442623 instructions/second 060608 112116 Processing linksByURL: Merged to new DB containing 15 records in 0.0030 seconds 060608 112116 Processing linksByURL: Merged 5000.0 records/second 060608 112116 Processing linksByMD5: Sorted 15 instructions in 0.0070 seconds. 060608 112116 Processing linksByMD5: Sorted 2142.8571428571427 instructions/second 060608 112116 Processing linksByMD5: Merged to new DB containing 15 records in 0.0010 seconds 060608 112116 Processing linksByMD5: Merged 15000.0 records/second 060608 112117 Update finished 060608 112117 FetchListTool started 060608 112117 Overall processing: Sorted 0 entries in 0.0 seconds. 060608 112117 Overall processing: Sorted NaN entries/second 060608 112117 FetchListTool completed 060608 112117 logging at INFO 060608 112118 Updating /home/mholt/Desktop/nutch-0.7.2/crawl/db 060608 112118 Updating for /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112117 060608 112118 Finishing update 060608 112118 Update finished 060608 112118 FetchListTool started 060608 112118 Overall processing: Sorted 0 entries in 0.0 seconds. 060608 112118 Overall processing: Sorted NaN entries/second 060608 112118 FetchListTool completed 060608 112118 logging at INFO 060608 112119 Updating /home/mholt/Desktop/nutch-0.7.2/crawl/db 060608 112119 Updating for /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112118 060608 112119 Finishing update 060608 112119 Update finished 060608 112119 FetchListTool started 060608 112119 Overall processing: Sorted 0 entries in 0.0 seconds. 060608 112119 Overall processing: Sorted NaN entries/second 060608 112119 FetchListTool completed 060608 112119 logging at INFO 060608 112120 Updating /home/mholt/Desktop/nutch-0.7.2/crawl/db 060608 112120 Updating for /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112119 060608 112120 Finishing update 060608 112120 Update finished 060608 112120 FetchListTool started 060608 112121 Overall processing: Sorted 0 entries in 0.0 seconds. 060608 112121 Overall processing: Sorted NaN entries/second 060608 112121 FetchListTool completed 060608 112121 logging at INFO 060608 112122 Updating /home/mholt/Desktop/nutch-0.7.2/crawl/db 060608 112122 Updating for /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112120 060608 112122 Finishing update 060608 112122 Update finished 060608 112122 FetchListTool started 060608 112122 Overall processing: Sorted 0 entries in 0.0 seconds. 060608 112122 Overall processing: Sorted NaN entries/second 060608 112122 FetchListTool completed 060608 112122 logging at INFO 060608 112123 Updating /home/mholt/Desktop/nutch-0.7.2/crawl/db 060608 112123 Updating for /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112122 060608 112123 Finishing update 060608 112123 Update finished 060608 112123 FetchListTool started 060608 112123 Overall processing: Sorted 0 entries in 0.0 seconds. 060608 112123 Overall processing: Sorted NaN entries/second 060608 112123 FetchListTool completed 060608 112123 logging at INFO 060608 112124 Updating /home/mholt/Desktop/nutch-0.7.2/crawl/db 060608 112124 Updating for /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112123 060608 112124 Finishing update 060608 112124 Update finished 060608 112124 FetchListTool started 060608 112124 Overall processing: Sorted 0 entries in 0.0 seconds. 060608 112124 Overall processing: Sorted NaN entries/second 060608 112124 FetchListTool completed 060608 112124 logging at INFO 060608 112125 Updating /home/mholt/Desktop/nutch-0.7.2/crawl/db 060608 112125 Updating for /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112124 060608 112125 Finishing update 060608 112125 Update finished 060608 112125 FetchListTool started 060608 112126 Overall processing: Sorted 0 entries in 0.0 seconds. 060608 112126 Overall processing: Sorted NaN entries/second 060608 112126 FetchListTool completed 060608 112126 logging at INFO 060608 112127 Updating /home/mholt/Desktop/nutch-0.7.2/crawl/db 060608 112127 Updating for /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112125 060608 112127 Finishing update 060608 112127 Update finished 060608 112127 Updating /home/mholt/Desktop/nutch-0.7.2/crawl/segments from /home/mholt/Desktop/nutch-0.7.2/crawl/db 060608 112127 reading /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112100 060608 112127 reading /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112103 060608 112127 reading /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112117 060608 112127 reading /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112118 060608 112127 reading /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112119 060608 112127 reading /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112120 060608 112127 reading /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112122 060608 112127 reading /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112123 060608 112127 reading /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112124 060608 112127 reading /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112125 060608 112127 Sorting pages by url... 060608 112127 Getting updated scores and anchors from db... 060608 112127 Sorting updates by segment... 060608 112127 Updating segments... 060608 112127 updating /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112100 060608 112127 updating /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112103 060608 112127 Done updating /home/mholt/Desktop/nutch-0.7.2/crawl/segments from /home/mholt/Desktop/nutch-0.7.2/crawl/db 060608 112127 indexing segment: /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112100 060608 112127 * Opening segment 20060608112100 060608 112127 * Indexing segment 20060608112100 060608 112127 found resource common-terms.utf8 at file:/home/mholt/Desktop/nutch-0.7.2/conf/common-terms.utf8 060608 112127 * Optimizing index... 060608 112127 * Moving index to NFS if needed... 060608 112127 DONE indexing segment 20060608112100: total 1 records in 0.225 s (Infinity rec/s). 060608 112127 done indexing 060608 112127 indexing segment: /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112103 060608 112127 * Opening segment 20060608112103 060608 112127 * Indexing segment 20060608112103 060608 112127 * Optimizing index... 060608 112127 * Moving index to NFS if needed... 060608 112127 DONE indexing segment 20060608112103: total 5 records in 0.164 s (Infinity rec/s). 060608 112127 done indexing 060608 112127 indexing segment: /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112117 060608 112127 * Opening segment 20060608112117 060608 112127 * Indexing segment 20060608112117 060608 112127 * Optimizing index... 060608 112127 * Moving index to NFS if needed... 060608 112127 DONE indexing segment 20060608112117: total 0 records in 0.011 s (NaN rec/s). 060608 112127 done indexing 060608 112127 indexing segment: /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112118 060608 112127 * Opening segment 20060608112118 060608 112127 * Indexing segment 20060608112118 060608 112127 * Optimizing index... 060608 112127 * Moving index to NFS if needed... 060608 112127 DONE indexing segment 20060608112118: total 0 records in 0.015 s (NaN rec/s). 060608 112127 done indexing 060608 112127 indexing segment: /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112119 060608 112127 * Opening segment 20060608112119 060608 112127 * Indexing segment 20060608112119 060608 112127 * Optimizing index... 060608 112127 * Moving index to NFS if needed... 060608 112127 DONE indexing segment 20060608112119: total 0 records in 0.013 s (NaN rec/s). 060608 112127 done indexing 060608 112128 indexing segment: /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112120 060608 112128 * Opening segment 20060608112120 060608 112128 * Indexing segment 20060608112120 060608 112128 * Optimizing index... 060608 112128 * Moving index to NFS if needed... 060608 112128 DONE indexing segment 20060608112120: total 0 records in 0.066 s (NaN rec/s). 060608 112128 done indexing 060608 112128 indexing segment: /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112122 060608 112128 * Opening segment 20060608112122 060608 112128 * Indexing segment 20060608112122 060608 112128 * Optimizing index... 060608 112128 * Moving index to NFS if needed... 060608 112128 DONE indexing segment 20060608112122: total 0 records in 0.019 s (NaN rec/s). 060608 112128 done indexing 060608 112128 indexing segment: /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112123 060608 112128 * Opening segment 20060608112123 060608 112128 * Indexing segment 20060608112123 060608 112128 * Optimizing index... 060608 112128 * Moving index to NFS if needed... 060608 112128 DONE indexing segment 20060608112123: total 0 records in 0.0090 s (NaN rec/s). 060608 112128 done indexing 060608 112128 indexing segment: /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112124 060608 112128 * Opening segment 20060608112124 060608 112128 * Indexing segment 20060608112124 060608 112128 * Optimizing index... 060608 112128 * Moving index to NFS if needed... 060608 112128 DONE indexing segment 20060608112124: total 0 records in 0.0090 s (NaN rec/s). 060608 112128 done indexing 060608 112128 indexing segment: /home/mholt/Desktop/nutch-0.7.2/crawl/segments/20060608112125 060608 112128 * Opening segment 20060608112125 060608 112128 * Indexing segment 20060608112125 060608 112128 * Optimizing index... 060608 112128 * Moving index to NFS if needed... 060608 112128 DONE indexing segment 20060608112125: total 0 records in 0.0090 s (NaN rec/s). 060608 112128 done indexing 060608 112128 Reading url hashes... 060608 112128 Sorting url hashes... 060608 112128 Deleting url duplicates... 060608 112128 Deleted 0 url duplicates. 060608 112128 Reading content hashes... 060608 112128 Sorting content hashes... 060608 112128 Deleting content duplicates... 060608 112128 Deleted 1 content duplicates. 060608 112128 Duplicate deletion complete locally. Now returning to NFS... 060608 112128 DeleteDuplicates complete 060608 112128 Merging segment indexes... 060608 112128 crawl finished: crawl/ _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
