Hi dean is this sorted On Saturday, January 7, 2012, Dean Pullen <[email protected]> wrote: > Sorry, you did mean on solrindex - which I already do... > > On 07/01/2012 13:15, Dean Pullen wrote: > > The -linkdb param isn't in the invertlinks docs http://wiki.apache.org/nutch/bin/nutch_invertlinks > > (However it is in the solrindex docs) > > Adding it makes no difference to invertlinks. > > I think the problem is definitely with mergesegs, as opposed to invertlinks etc. > > Thanks again, > > Dean. > > On 06/01/2012 17:53, Lewis John Mcgibbney wrote: > > OK so now I think were at the bottom of it. If you wish to create a > linkdb in>= Nutch 1.4 you need to specifically pass the linkdb > parameter. This was implemented as not everyone wishes to create a > linkdb. > > Your invertlinks command should be passed as follows > > bin/nutch invertlinks path/you/wish/to/have/the/linkdb -dir > /path/to/segment/dirs > then > bin/nutch solrindex http://solrUrl path/to/crawldb -linkdb > path/to/linkdb -dir path/to/segment/dirs > > If you are not passing the -linkdb path/to/linkdb explicitly you will > be thrown an exception as the linkdb is treated as a segment directory > now. > > On Fri, Jan 6, 2012 at 5:17 PM, Dean Pullen<[email protected]> wrote: > > Only this: > > 2012-01-06 17:15:47,972 WARN mapred.JobClient - Use GenericOptionsParser > for parsing the arguments. Applications should implement Tool for the same. > 2012-01-06 17:15:48,692 WARN util.NativeCodeLoader - Unable to load > native-hadoop library for your platform... using builtin-java classes where > applicable > 2012-01-06 17:15:51,566 INFO crawl.LinkDb - LinkDb: starting at 2012-01-06 > 17:15:51 > 2012-01-06 17:15:51,567 INFO crawl.LinkDb - LinkDb: linkdb: > /opt/nutch_1_4/data/crawl/linkdb > 2012-01-06 17:15:51,567 INFO crawl.LinkDb - LinkDb: URL normalize: true > 2012-01-06 17:15:51,567 INFO crawl.LinkDb - LinkDb: URL filter: true > 2012-01-06 17:15:51,576 INFO crawl.LinkDb - LinkDb: adding segment: > file:/opt/nutch_1_4/data/crawl/segments/20120106171547 > 2012-01-06 17:15:51,721 ERROR crawl.LinkDb - LinkDb: > org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: > file:/opt/nutch_1_4/data/crawl/segments/20120106171547/parse_data > at > org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:190) > at > org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:44) > at > org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:201) > at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810) > at > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781) > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730) > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249) > at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:175) > at org.apache.nutch.crawl.LinkDb.run(LinkDb.java:290) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.nutch.crawl.LinkDb.main(LinkDb.java:255) > > 2012-01-06 17:15:52,714 INFO solr.SolrIndexer - SolrIndexer: starting at > 2012-01-06 17:15:52 > 2012-01-06 17:15:52,782 INFO indexer.IndexerMapReduce - IndexerMapReduce: > crawldb: /opt/nutch_1_4/data/crawl/crawldb > 2012-01-06 17:15:52,782 INFO indexer.IndexerMapReduce - IndexerMapReduce: > linkdb: /opt/nutch_1_4/data/crawl/linkdb >
-- *Lewis*
