After the error before (see post for config files: Local filesystem crawl
problem).
I got this error:
...
fetching file:///cygdrive/c/Temp
org.apache.nutch.protocol.file.FileError: File Error: 404
at
org.apache.nutch.protocol.file.File.getProtocolOutput(File.java:100)
at
org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:145)
fetch of file:///cygdrive/c/Temp failed with:
org.apache.nutch.protocol.file.Fil
eError: File Error: 404
Fetcher: done
....
I configured in a text file urls/urls.tct to crawl in the directory
file:///cygdrive/c/Temp
The folder exists, but it seems to don't find it, i expected the clawler to
find files inside that,
and fetch them...but it doesn't.
Thank you
Paolo
----- Original Message -----
From: "Paolo Mazzoni" <[EMAIL PROTECTED]>
To: <[email protected]>
Sent: Monday, August 11, 2008 11:02 AM
Subject: Local filesystem crawl problem
Hi all,
i'm very new to Nutch, and i'd like to setup a single search server.
I'm actually working on Windows Vista for testing purposes.
I was able to crawl some websites, but when i try to configure Nutch
to crawl the local file system i got the following error:
[EMAIL PROTECTED] /cygdrive/c/sviluppo/CVS/apps/nutch-0.9
$ bin/nutch crawl urls -dir crawl-localfs -depth 3 -topN 50
crawl started in: crawl-localfs
rootUrlDir = urls
threads = 10
depth = 3
topN = 50
Injector: starting
Injector: crawlDb: crawl-localfs/crawldb
Injector: urlDir: urls
Injector: Converting injected urls to crawl db entries.
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604)
at org.apache.nutch.crawl.Injector.inject(Injector.java:162)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:115)
-----
My crawl-urlfilter.txt is:
# The url filter file used by the crawl command.
# Better for intranet crawling.
# Be sure to change MY.DOMAIN.NAME to your domain name.
# Each non-comment, non-blank line contains a regular expression
# prefixed by '+' or '-'. The first matching pattern in the file
# determines whether a URL is included or ignored. If no pattern
# matches, the URL is ignored.
# skip file:, ftp:, & mailto: urls
#-^(file|ftp|mailto):
-^(http|ftp|mailto):
# skip image and other suffixes we can't yet parse
-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|MOV|exe|jpeg|JPEG|bmp|BMP)$
# skip URLs containing certain characters as probable queries, etc.
[EMAIL PROTECTED]
# skip URLs with slash-delimited segment that repeats 3+ times, to break
loops
-.*(/.+?)/.*?\1/.*?\1/
# accept hosts in MY.DOMAIN.NAME
#+^http://([a-z0-9]*\.)*MY.DOMAIN.NAME/
# skip everything else
#-.
+.*
----
And my nutch-site.xml is (and i have a lot of dobts on this):
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>plugin.includes</name>
<value>protocol-file|urlfilter-regex|parse-(xml|text|html|js|pdf)|index-basic|query-(basic|site|url)</value>
</property>
<property>
<name>file.content.limit</name> <value>-1</value>
</property>
</configuration>
----
Under urls/ directory i hace a txt file ulrs.txt containing the folders to
index:
file:///cygdrive/c/Temp/
No virus found in this incoming message.
Checked by AVG - http://www.avg.com
Version: 8.0.138 / Virus Database: 270.6.0/1603 - Release Date: 10/08/2008
18.13
