Many thanks Remi.
Finally, after un reboot og the computer (I send my question just before
leaving my desk), Nutch started to crawl (amazing :))) )
But now, during the crawl process, I got that :
-----
LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222161934
LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120223093525
LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222153642
LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222154459
Exception in thread "main"
org.apache.hadoop.mapred.InvalidInputException: Input path does not
exist:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222160234/parse_data
Input path does not exist:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222160609/parse_data
Input path does not exist:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222153805/parse_data
Input path does not exist:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222155532/parse_data
Input path does not exist:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222160132/parse_data
Input path does not exist:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222153642/parse_data
Input path does not exist:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222154459/parse_data
at
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:190)
at
org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:44)
at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:201)
at
org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249)
at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:175)
at org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:149)
at org.apache.nutch.crawl.Crawl.run(Crawl.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
-----
and nothing special in the logs :
last lines are :
2012-02-23 09:46:42,524 INFO crawl.CrawlDb - CrawlDb update: finished
at 2012-02-23 09:46:42, elapsed: 00:00:01
2012-02-23 09:46:42,590 INFO crawl.LinkDb - LinkDb: starting at
2012-02-23 09:46:42
2012-02-23 09:46:42,591 INFO crawl.LinkDb - LinkDb: linkdb: crawl/linkdb
2012-02-23 09:46:42,591 INFO crawl.LinkDb - LinkDb: URL normalize: true
2012-02-23 09:46:42,591 INFO crawl.LinkDb - LinkDb: URL filter: true
2012-02-23 09:46:42,593 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120223093220
2012-02-23 09:46:42,593 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222160234
2012-02-23 09:46:42,593 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120223093302
2012-02-23 09:46:42,594 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222160609
2012-02-23 09:46:42,594 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222153805
2012-02-23 09:46:42,594 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222155532
2012-02-23 09:46:42,594 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120223094427
2012-02-23 09:46:42,594 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120223093618
2012-02-23 09:46:42,595 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120223094552
2012-02-23 09:46:42,595 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120223094500
2012-02-23 09:46:42,595 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222160132
2012-02-23 09:46:42,595 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120223093649
2012-02-23 09:46:42,596 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120223093210
2012-02-23 09:46:42,596 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222161934
2012-02-23 09:46:42,596 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120223093525
2012-02-23 09:46:42,596 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222153642
2012-02-23 09:46:42,597 INFO crawl.LinkDb - LinkDb: adding segment:
file:/home/daniel/Bureau/apache-nutch-1.4-bin/runtime/local/crawl/segments/20120222154459
On 22/02/2012 16:36, remi tassing wrote:
Hey Daniel,
You can find more output log in logs/Hadoop files
Remi
On Wednesday, February 22, 2012, Daniel Bourrion<
[email protected]> wrote:
Hi.
I'm a french librarian (that explains the bad english coming now... :) )
Newbie on Nutch, that looks exactly what i'm searching for (an opensource
solution that should crawl our specific domaine and have it's crawl results
pushed into Solr).
I've install a test nutch using http://wiki.apache.org/nutch/NutchTutorial
Got an error but I don't really know it nor understand where to try to
correct what causes that.
Here's a copy of the error messages - any help welcome.
Best
--------------------------------------------------
daniel@daniel-linux:~/Bureau/apache-nutch-1.4-bin/runtime/local$
bin/nutch crawl urls -dir crawl -depth 3 -topN 5
solrUrl is not set, indexing will be skipped...
crawl started in: crawl
rootUrlDir = urls
threads = 10
depth = 3
solrUrl=null
topN = 5
Injector: starting at 2012-02-22 16:06:04
Injector: crawlDb: crawl/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Injector: Merging injected urls into crawl db.
Injector: finished at 2012-02-22 16:06:06, elapsed: 00:00:02
Generator: starting at 2012-02-22 16:06:06
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: true
Generator: normalizing: true
Generator: topN: 5
Generator: jobtracker is 'local', generating exactly one partition.
Generator: Partitioning selected urls for politeness.
Generator: segment: crawl/segments/20120222160609
Generator: finished at 2012-02-22 16:06:10, elapsed: 00:00:03
Fetcher: Your 'http.agent.name' value should be listed first in
'http.robots.agents' property.
Fetcher: starting at 2012-02-22 16:06:10
Fetcher: segment: crawl/segments/20120222160609
Using queue mode : byHost
Fetcher: threads: 10
Fetcher: time-out divisor: 2
QueueFeeder finished: total 2 records + hit by time limit :0
Using queue mode : byHost
Using queue mode : byHost
Using queue mode : byHost
fetching http://bu.univ-angers.fr/
Using queue mode : byHost
Using queue mode : byHost
fetching http://www.face-ecran.fr/
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=2
Using queue mode : byHost
-finishing thread FetcherThread, activeThreads=2
Using queue mode : byHost
-finishing thread FetcherThread, activeThreads=2
Using queue mode : byHost
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=2
Using queue mode : byHost
Using queue mode : byHost
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=2
Fetcher: throughput threshold: -1
Fetcher: throughput threshold retries: 5
-activeThreads=2, spinWaiting=0, fetchQueues.totalSize=0
-finishing thread FetcherThread, activeThreads=1
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0
-activeThreads=0
Fetcher: finished at 2012-02-22 16:06:13, elapsed: 00:00:03
ParseSegment: starting at 2012-02-22 16:06:13
ParseSegment: segment: crawl/segments/20120222160609
Parsing: http://bu.univ-angers.fr/
Parsing: http://www.face-ecran.fr/
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
at org.apache.nutch.parse.ParseSegment.parse(ParseSegment.java:157)
at org.apache.nutch.crawl.Crawl.run(Crawl.java:138)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
--------------------------------------------------
--
Avec mes salutations les plus cordiales.
__
Daniel Bourrion, conservateur des bibliothèques
Responsable de la bibliothèque numérique
Ligne directe : 02.44.68.80.50
SCD Université d'Angers - http://bu.univ-angers.fr
Bu Saint Serge - 57 Quai Félix Faure - 49100 Angers cedex
***********************************
" Et par le pouvoir d'un mot
Je recommence ma vie "
Paul Eluard
***********************************
blog perso : http://archives.face-ecran.fr/
--
Avec mes salutations les plus cordiales.
__
Daniel Bourrion, conservateur des bibliothèques
Responsable de la bibliothèque numérique
Ligne directe : 02.44.68.80.50
SCD Université d'Angers - http://bu.univ-angers.fr
Bu Saint Serge - 57 Quai Félix Faure - 49100 Angers cedex
***********************************
" Et par le pouvoir d'un mot
Je recommence ma vie "
Paul Eluard
***********************************
blog perso : http://archives.face-ecran.fr/
