Hi.
I'm a french librarian (that explains the bad english coming now... :) )
Newbie on Nutch, that looks exactly what i'm searching for (an
opensource solution that should crawl our specific domaine and have it's
crawl results pushed into Solr).
I've install a test nutch using http://wiki.apache.org/nutch/NutchTutorial
Got an error but I don't really know it nor understand where to try to
correct what causes that.
Here's a copy of the error messages - any help welcome.
Best
--------------------------------------------------
daniel@daniel-linux:~/Bureau/apache-nutch-1.4-bin/runtime/local$
bin/nutch crawl urls -dir crawl -depth 3 -topN 5
solrUrl is not set, indexing will be skipped...
crawl started in: crawl
rootUrlDir = urls
threads = 10
depth = 3
solrUrl=null
topN = 5
Injector: starting at 2012-02-22 16:06:04
Injector: crawlDb: crawl/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Injector: Merging injected urls into crawl db.
Injector: finished at 2012-02-22 16:06:06, elapsed: 00:00:02
Generator: starting at 2012-02-22 16:06:06
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: topN: 5
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: crawl/segments/20120222160609
Generator: finished at 2012-02-22 16:06:10, elapsed: 00:00:03
Fetcher: Your 'http.agent.name' value should be listed first in
'http.robots.agents' property.
Fetcher: starting at 2012-02-22 16:06:10
Fetcher: segment: crawl/segments/20120222160609
Using queue mode : byHost
Fetcher: threads: 10
Fetcher: time-out divisor: 2
QueueFeeder finished: total 2 records + hit by time limit :0
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
fetching http://bu.univ-angers.fr/
Using queue mode : byHost
Using queue mode : byHost
fetching http://www.face-ecran.fr/
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=2
Using queue mode : byHost
-finishing thread FetcherThread, activeThreads=2
Using queue mode : byHost
-finishing thread FetcherThread, activeThreads=2
Using queue mode : byHost
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=2
Using queue mode : byHost
Using queue mode : byHost
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=2
Fetcher: throughput threshold: -1
Fetcher: throughput threshold retries: 5
-activeThreads=2, spinWaiting=0, fetchQueues.totalSize=0
-finishing thread FetcherThread, activeThreads=1
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0
Fetcher: finished at 2012-02-22 16:06:13, elapsed: 00:00:03
ParseSegment: starting at 2012-02-22 16:06:13
ParseSegment: segment: crawl/segments/20120222160609
Parsing: http://bu.univ-angers.fr/
Parsing: http://www.face-ecran.fr/
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.nutch.parse.ParseSegment.parse(ParseSegment.java:157)
at org.apache.nutch.crawl.Crawl.run(Crawl.java:138)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
--------------------------------------------------
--
Avec mes salutations les plus cordiales.
__
Daniel Bourrion, conservateur des bibliothèques
Responsable de la bibliothèque numérique
Ligne directe : 02.44.68.80.50
SCD Université d'Angers - http://bu.univ-angers.fr
Bu Saint Serge - 57 Quai Félix Faure - 49100 Angers cedex
***********************************
" Et par le pouvoir d'un mot
Je recommence ma vie "
Paul Eluard
***********************************
blog perso : http://archives.face-ecran.fr/
