The -linkdb param isn't in the invertlinks docs
http://wiki.apache.org/nutch/bin/nutch_invertlinks
(However it is in the solrindex docs)
Adding it makes no difference to invertlinks.
I think the problem is definitely with mergesegs, as opposed to
invertlinks etc.
Thanks again,
Dean.
On 06/01/2012 17:53, Lewis John Mcgibbney wrote:
OK so now I think were at the bottom of it. If you wish to create a
linkdb in>= Nutch 1.4 you need to specifically pass the linkdb
parameter. This was implemented as not everyone wishes to create a
linkdb.
Your invertlinks command should be passed as follows
bin/nutch invertlinks path/you/wish/to/have/the/linkdb -dir
/path/to/segment/dirs
then
bin/nutch solrindex http://solrUrl path/to/crawldb -linkdb
path/to/linkdb -dir path/to/segment/dirs
If you are not passing the -linkdb path/to/linkdb explicitly you will
be thrown an exception as the linkdb is treated as a segment directory
now.
On Fri, Jan 6, 2012 at 5:17 PM, Dean Pullen<[email protected]> wrote:
Only this:
2012-01-06 17:15:47,972 WARN mapred.JobClient - Use GenericOptionsParser
for parsing the arguments. Applications should implement Tool for the same.
2012-01-06 17:15:48,692 WARN util.NativeCodeLoader - Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
2012-01-06 17:15:51,566 INFO crawl.LinkDb - LinkDb: starting at 2012-01-06
17:15:51
2012-01-06 17:15:51,567 INFO crawl.LinkDb - LinkDb: linkdb:
/opt/nutch_1_4/data/crawl/linkdb
2012-01-06 17:15:51,567 INFO crawl.LinkDb - LinkDb: URL normalize: true
2012-01-06 17:15:51,567 INFO crawl.LinkDb - LinkDb: URL filter: true
2012-01-06 17:15:51,576 INFO crawl.LinkDb - LinkDb: adding segment:
file:/opt/nutch_1_4/data/crawl/segments/20120106171547
2012-01-06 17:15:51,721 ERROR crawl.LinkDb - LinkDb:
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:
file:/opt/nutch_1_4/data/crawl/segments/20120106171547/parse_data
at
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:190)
at
org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:44)
at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:201)
at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249)
at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:175)
at org.apache.nutch.crawl.LinkDb.run(LinkDb.java:290)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.LinkDb.main(LinkDb.java:255)
2012-01-06 17:15:52,714 INFO solr.SolrIndexer - SolrIndexer: starting at
2012-01-06 17:15:52
2012-01-06 17:15:52,782 INFO indexer.IndexerMapReduce - IndexerMapReduce:
crawldb: /opt/nutch_1_4/data/crawl/crawldb
2012-01-06 17:15:52,782 INFO indexer.IndexerMapReduce - IndexerMapReduce:
linkdb: /opt/nutch_1_4/data/crawl/linkdb
2012-01-06 17:15:52,782 INFO indexer.IndexerMapReduce - IndexerMapReduces:
adding segment: /opt/nutch_1_4/data/crawl/segments/20120106171547
2012-01-06 17:15:53,000 ERROR solr.SolrIndexer -
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:
file:/opt/nutch_1_4/data/crawl/segments/20120106171547/crawl_parse
Input path does not exist:
file:/opt/nutch_1_4/data/crawl/segments/20120106171547/parse_data
Input path does not exist:
file:/opt/nutch_1_4/data/crawl/segments/20120106171547/parse_text
2012-01-06 17:15:54,027 INFO crawl.CrawlDbReader - CrawlDb dump: starting
2012-01-06 17:15:54,028 INFO crawl.CrawlDbReader - CrawlDb db:
/opt/nutch_1_4/data/crawl/crawldb/
2012-01-06 17:15:54,212 WARN mapred.JobClient - Use GenericOptionsParser
for parsing the arguments. Applications should implement Tool for the same.
2012-01-06 17:15:55,603 INFO crawl.CrawlDbReader - CrawlDb dump: done
