I fixed the errors, thanks. On Sat, Aug 18, 2012 at 1:33 AM, Robert Irribarren <[email protected]>wrote:
> And here is my hadoop.log > 2012-08-18 08:30:13,069 INFO solr.SolrIndexerJob - SolrIndexerJob: > starting > 2012-08-18 08:30:13,658 INFO plugin.PluginRepository - Plugins: looking > in: /usr/share/nutch/runtime/local/plugins > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - Plugin > Auto-activation mode: [true] > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - Registered Plugins: > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - the nutch > core extension points (nutch-extensionpoints) > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - Basic URL > Normalizer (urlnormalizer-basic) > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - Html Parse > Plug-in (parse-html) > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - Basic > Indexing Filter (index-basic) > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - HTTP > Framework (lib-http) > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - > Pass-through URL Normalizer (urlnormalizer-pass) > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - Regex URL > Filter (urlfilter-regex) > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - Http > Protocol Plug-in (protocol-http) > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - Regex URL > Normalizer (urlnormalizer-regex) > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - Tika > Parser Plug-in (parse-tika) > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - OPIC > Scoring Plug-in (scoring-opic) > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - CyberNeko > HTML Parser (lib-nekohtml) > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - Anchor > Indexing Filter (index-anchor) > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - Regex URL > Filter Framework (lib-regex-filter) > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - Registered > Extension-Points: > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - Nutch URL > Normalizer (org.apache.nutch.net.URLNormalizer) > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - Nutch > Protocol (org.apache.nutch.protocol.Protocol) > 2012-08-18 08:30:13,866 INFO plugin.PluginRepository - Parse > Filter (org.apache.nutch.parse.ParseFilter) > 2012-08-18 08:30:13,867 INFO plugin.PluginRepository - Nutch URL > Filter (org.apache.nutch.net.URLFilter) > 2012-08-18 08:30:13,867 INFO plugin.PluginRepository - Nutch > Indexing Filter (org.apache.nutch.indexer.IndexingFilter) > 2012-08-18 08:30:13,867 INFO plugin.PluginRepository - Nutch > Content Parser (org.apache.nutch.parse.Parser) > 2012-08-18 08:30:13,867 INFO plugin.PluginRepository - Nutch > Scoring (org.apache.nutch.scoring.ScoringFilter) > 2012-08-18 08:30:13,881 INFO indexer.IndexingFilters - Adding > org.apache.nutch.indexer.basic.BasicIndexingFilter > 2012-08-18 08:30:13,883 INFO anchor.AnchorIndexingFilter - Anchor > deduplication is: off > 2012-08-18 08:30:13,883 INFO indexer.IndexingFilters - Adding > org.apache.nutch.indexer.anchor.AnchorIndexingFilter > 2012-08-18 08:30:14,946 WARN util.NativeCodeLoader - Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > 2012-08-18 08:30:15,960 INFO mapreduce.GoraRecordReader - > gora.buffer.read.limit = 10000 > 2012-08-18 08:30:16,091 INFO solr.SolrMappingReader - source: content > dest: content > 2012-08-18 08:30:16,091 INFO solr.SolrMappingReader - source: site dest: > site > 2012-08-18 08:30:16,091 INFO solr.SolrMappingReader - source: title dest: > title > 2012-08-18 08:30:16,091 INFO solr.SolrMappingReader - source: host dest: > host > 2012-08-18 08:30:16,092 INFO solr.SolrMappingReader - source: segment > dest: segment > 2012-08-18 08:30:16,092 INFO solr.SolrMappingReader - source: boost dest: > boost > 2012-08-18 08:30:16,092 INFO solr.SolrMappingReader - source: digest > dest: digest > 2012-08-18 08:30:16,092 INFO solr.SolrMappingReader - source: tstamp > dest: tstamp > 2012-08-18 08:30:16,094 INFO indexer.IndexingFilters - Adding > org.apache.nutch.indexer.basic.BasicIndexingFilter > 2012-08-18 08:30:16,094 INFO anchor.AnchorIndexingFilter - Anchor > deduplication is: off > 2012-08-18 08:30:16,094 INFO indexer.IndexingFilters - Adding > org.apache.nutch.indexer.anchor.AnchorIndexingFilter > 2012-08-18 08:30:16,957 INFO solr.SolrWriter - Adding 36 documents > 2012-08-18 08:30:19,859 INFO solr.SolrIndexerJob - SolrIndexerJob: done. > > > > On Sat, Aug 18, 2012 at 1:09 AM, Robert Irribarren > <[email protected]>wrote: > >> WebTable statistics start >> Statistics for WebTable: >> min score: 0.0 >> status 2 (status_fetched): 1053 >> jobs: {db_stats-job_local_0001={jobID=job_local_0001, jobName=db_stats, >> counters={File Input Format Counters ={BYTES_READ=0}, Map-Reduce >> Framework={MAP_OUTPUT_MATERIALIZED_BYTES=211, MAP_INPUT_RECORDS=1234, >> REDUCE_SHUFFLE_BYTES=0, SPILLED_RECORDS=24, MAP_OUTPUT_BYTES=65418, >> COMMITTED_HEAP_BYTES=504635392, CPU_MILLISECONDS=0, SPLIT_RAW_BYTES=1046, >> COMBINE_INPUT_RECORDS=4936, REDUCE_INPUT_RECORDS=12, >> REDUCE_INPUT_GROUPS=12, COMBINE_OUTPUT_RECORDS=12, PHYSICAL_MEMORY_BYTES=0, >> REDUCE_OUTPUT_RECORDS=12, VIRTUAL_MEMORY_BYTES=0, MAP_OUTPUT_RECORDS=4936}, >> FileSystemCounters={FILE_BYTES_READ=878225, FILE_BYTES_WRITTEN=991145}, >> File Output Format Counters ={BYTES_WRITTEN=375}}}} >> retry 0: 1233 >> retry 1: 1 >> TOTAL urls: 1234 >> status 4 (status_redir_temp): 32 >> status 5 (status_redir_perm): 47 >> max score: 1.0 >> status 34 (status_retry): 16 >> status 3 (status_gone): 17 >> status 0 (null): 69 >> avg score: 0.01614992 >> WebTable statistics: done >> min score: 0.0 >> status 2 (status_fetched): 1053 >> jobs: {db_stats-job_local_0001={jobID=job_local_0001, jobName=db_stats, >> counters={File Input Format Counters ={BYTES_READ=0}, Map-Reduce >> Framework={MAP_OUTPUT_MATERIALIZED_BYTES=211, MAP_INPUT_RECORDS=1234, >> REDUCE_SHUFFLE_BYTES=0, SPILLED_RECORDS=24, MAP_OUTPUT_BYTES=65418, >> COMMITTED_HEAP_BYTES=504635392, CPU_MILLISECONDS=0, SPLIT_RAW_BYTES=1046, >> COMBINE_INPUT_RECORDS=4936, REDUCE_INPUT_RECORDS=12, >> REDUCE_INPUT_GROUPS=12, COMBINE_OUTPUT_RECORDS=12, PHYSICAL_MEMORY_BYTES=0, >> REDUCE_OUTPUT_RECORDS=12, VIRTUAL_MEMORY_BYTES=0, MAP_OUTPUT_RECORDS=4936}, >> FileSystemCounters={FILE_BYTES_READ=878225, FILE_BYTES_WRITTEN=991145}, >> File Output Format Counters ={BYTES_WRITTEN=375}}}} >> retry 0: 1233 >> retry 1: 1 >> TOTAL urls: 1234 >> status 4 (status_redir_temp): 32 >> status 5 (status_redir_perm): 47 >> max score: 1.0 >> status 34 (status_retry): 16 >> status 3 (status_gone): 17 >> status 0 (null): 69 >> avg score: 0.01614992 >> >> >> This is what the db says but its not really what i see on my solr. >> Perhaps I didn't set my solr directory somewhere? Please help >> >> >> On Sat, Aug 18, 2012 at 12:59 AM, Robert Irribarren <[email protected] >> > wrote: >> >>> Update : I get this after im done crawling >>> >>> Parsing http://www.brainpop.co.uk/ >>> Exception in thread "main" java.lang.RuntimeException: job failed: >>> name=parse, jobid=job_local_0004 >>> at >>> org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:47) >>> at org.apache.nutch.parse.ParserJob.run(ParserJob.java:249) >>> at org.apache.nutch.crawl.Crawler.runTool(Crawler.java:68) >>> at org.apache.nutch.crawl.Crawler.run(Crawler.java:171) >>> at org.apache.nutch.crawl.Crawler.run(Crawler.java:250) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>> at org.apache.nutch.crawl.Crawler.main(Crawler.java:257) >>> >>> >>> >>> On Sat, Aug 18, 2012 at 12:30 AM, Robert Irribarren < >>> [email protected]> wrote: >>> >>>> I actually didnt have it specified, I now put this in the >>>> nutch-site.xml looks like this. >>>> >>>> <?xml version="1.0"?> >>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> >>>> >>>> <!-- Put site-specific property overrides in this file. --> >>>> >>>> <configuration> >>>> <property> >>>> <name>http.agent.name</name> >>>> <value>Balsa Crawler</value> >>>> </property> >>>> >>>> <property> >>>> <name>db.ignore.external.links</name> >>>> <value>false</value> >>>> <description>If true, outlinks leading from a page to external hosts >>>> will be ignored. This is an effective way to limit the crawl to >>>> include >>>> only initially injected hosts, without creating complex URLFilters. >>>> </description> >>>> </property> >>>> >>>> <property> >>>> <name>storage.data.store.class</name> >>>> <value>org.apache.gora.sql.store.SqlStore</value> >>>> <description>The Gora DataStore class for storing and retrieving data. >>>> Currently the following stores are available: .. >>>> </description> >>>> </property> >>>> >>>> </configuration> >>>> >>>> >>>> >>>> On Sat, Aug 18, 2012 at 12:15 AM, Stefan Scheffler < >>>> [email protected]> wrote: >>>> >>>>> Did you set db.ignore.external in *conf/nutch-site.xml*? >>>>> This avoids that external links are fetched. >>>>> Another problem could be, that the robots.txt of the servers prevents >>>>> the crawler from fetching. >>>>> you can check this with *bin/nutch readdb*. There you see, if the >>>>> sites are really fetched >>>>> regards >>>>> Stefan >>>>> >>>>> Am 18.08.2012 09:07, schrieb Robert Irribarren: >>>>> >>>>> I run this >>>>>> nutch inject urls >>>>>> nutch generate >>>>>> bin/nutch crawl urls -depth 3 -topN 100 >>>>>> bin/nutch solrindex http://127.0.0.1:8983/solr/ -reindex >>>>>> echo Crawling completed >>>>>> dir >>>>>> >>>>>> then I see alot of urls being fetched during the crawl phase. >>>>>> When I run the solrindex it doesn't add all the urls i see when it >>>>>> says >>>>>> fetching >>>>>> >>>>>> 54 URLs in 5 queues >>>>>> fetching http://www.tarpits.org/join-us >>>>>> fetching >>>>>> http://www.leonisadobemuseum.**org/history-leonis.asp<http://www.leonisadobemuseum.org/history-leonis.asp> >>>>>> fetching >>>>>> http://az.wikipedia.org/wiki/**Quercus_prinus<http://az.wikipedia.org/wiki/Quercus_prinus> >>>>>> >>>>>> It doesn't add wikipedia nor the others. >>>>>> >>>>>> ADDITIONAL INFO : >>>>>> My regex-urlfilter.txt >>>>>> # skip file: ftp: and mailto: urls >>>>>> -^(file|ftp|mailto): >>>>>> >>>>>> # skip image and other suffixes we can't yet parse >>>>>> # for a more extensive coverage use the urlfilter-suffix plugin >>>>>> -\.(gif|GIF|jpg|JPG|png|PNG|**ico|ICO|css|CSS|sit|SIT|eps|** >>>>>> EPS|wmf|WMF|zip|ZIP|ppt|PPT|**mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|** >>>>>> tgz|TGZ|mov|MOV|exe|EXE|jpeg|**JPEG|bmp|BMP|js|JS)$ >>>>>> >>>>>> # skip URLs containing certain characters as probable queries, etc. >>>>>> -[?*!@=] >>>>>> >>>>>> # skip URLs with slash-delimited segment that repeats 3+ times, to >>>>>> break >>>>>> loops >>>>>> -.*(/[^/]+)/[^/]+\1/[^/]+\1/ >>>>>> >>>>>> # accept anything else >>>>>> +. >>>>>> ##############################**##############################**##### >>>>>> >>>>>> ADDITIONAL INFO : Running on solr 4.0 nutch 2.0 >>>>>> >>>>>> >>>>> >>>> >>> >> >
