I've tried crawling with nutch-1.6-SNAPSHOT.jar and got the following
error:
[root@w7sp1-x64 nutch]# bin/nutch crawl urls -dir crawl -depth 3 -topN 5
solrUrl is not set, indexing will be skipped...
crawl started in: crawl
rootUrlDir = urls
threads = 10
depth = 3
solrUrl=null
topN = 5
Injector: starting at 2012-09-07 08:41:06
Injector: crawlDb: crawl/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Injector: Merging injected urls into crawl db.
Injector: finished at 2012-09-07 08:41:21, elapsed: 00:00:14
Generator: starting at 2012-09-07 08:41:21
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: topN: 5
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: crawl/segments/20120907084129
Generator: finished at 2012-09-07 08:41:36, elapsed: 00:00:15
Fetcher: Your 'http.agent.name' value should be listed first in
'http.robots.agents' property.
Fetcher: starting at 2012-09-07 08:41:36
Fetcher: segment: crawl/segments/20120907084129
Using queue mode : byHost
Fetcher: threads: 10
Fetcher: time-out divisor: 2
QueueFeeder finished: total 1 records + hit by time limit :0
Exception in thread "main" java.io.IOException: Job failed!
at
org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1265)
at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1332)
at org.apache.nutch.crawl.Crawl.run(Crawl.java:136)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
I then tried to crawl with 1.5.1 (which was successful) and INDEX with
1.6-SNAPSHOT. I got this error:
[root@w7sp1-x64 nutch]# bin/nutch solrindex
http://127.0.0.1:8080/solr/core2 crawl/crawldb -linkdb crawl/linkdb
SolrIndexer: starting at 2012-09-07 09:05:21
SolrIndexer: deleting gone documents: false
SolrIndexer: URL filtering: false
SolrIndexer: URL normalizing: false
org.apache.solr.common.SolrException: Not Found
Not Found
request: http://127.0.0.1:8080/solr/core2/update
> -----Original Message-----
> From: Fournier, Danny G [mailto:[email protected]]
> Sent: September 6, 2012 4:15 PM
> To: [email protected]
> Subject: Errors when indexing to Solr
>
> I'm getting two different errors while trying to index Nutch crawls to
> Solr. I'm running with:
>
> - CentOS 6.3 VM (Virtualbox) (in host Windows XP)
> - Solr 3.6.1
> - Nutch 1.5.1
>
> It would seem that NUTCH-1251 comes rather close to solving my issue?
> Which would mean that I would have to compile Nutch 1.6 to fix this?
>
> Error #1 - When indexing directly to Solr
> ------------------------------------------------
> Command: bin/nutch crawl urls -solr http://localhost:8080/solr/core2
> -depth 3 -topN 5
>
> Error: Exception in thread "main" java.io.IOException:
> org.apache.solr.client.solrj.SolrServerException: Error executing
query
>
> SolrIndexer: starting at 2012-09-06 14:30:11
> Indexing 8 documents
> java.io.IOException: Job failed!
> SolrDeleteDuplicates: starting at 2012-09-06 14:30:55
> SolrDeleteDuplicates: Solr url: http://localhost:8080/solr/core2
> Exception in thread "main" java.io.IOException:
> org.apache.solr.client.solrj.SolrServerException: Error executing
> query
> at
>
org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getSp
> lits(SolrDeleteDuplicates.java:200)
> at
> org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:989)
> at
> org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:981)
> at
> org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:897)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInform
> atio
> n.java:1121)
> at
>
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
> at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:824)
> at
> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1261)
> at
>
org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDupli
> cates.java:373)
> at
>
org.apache.nutch.indexer.solr.SolrDeleteDuplicates.dedup(SolrDeleteDupli
> cates.java:353)
> at org.apache.nutch.crawl.Crawl.run(Crawl.java:153)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
> Caused by: org.apache.solr.client.solrj.SolrServerException: Error
> executing query
> at
>
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.j
> ava:95)
> at
> org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:118)
> at
>
org.apache.nutch.indexer.solr.SolrDeleteDuplicates$SolrInputFormat.getSp
> lits(SolrDeleteDuplicates.java:198)
> ... 16 more
> Caused by: org.apache.solr.common.SolrException: Not Found
>
> Not Found
>
> request: http://localhost:8080/solr/core2/select?q=id:[* TO
> *]&fl=id&rows=1&wt=javabin&version=2
> at
>
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(Commons
> H
> ttpSolrServer.java:430)
> at
>
org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(Commons
> H
> ttpSolrServer.java:244)
> at
>
org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.j
> ava:89)
> ... 18 more
>
> Error #2 - When indexing post-crawl
> --------------------------------------------
> Command: bin/nutch solrindex http://localhost:8080/solr/core2
> crawl/crawldb -linkdb crawl/linkdb
>
> Error: org.apache.solr.common.SolrException: Not Found
>
> SolrIndexer: starting at 2012-09-06 15:39:24
> org.apache.solr.common.SolrException: Not Found
>
> Not Found
>
> request: http://localhost:8080/solr/core2/update
>
>
> Regards,
>
> Dan
