Hi, These day I follow the Nutch totur: http://wiki.apache.org/nutch/NutchTutorial, but I always get the error message as follows:
MaohuaLiu-MacBook-Pro:apache-nutch-1.6 carya$ bin/crawl urls/seed.txt TestCrawl http://localhost:8983/solr/ 2 Injector: starting at 2013-04-23 22:00:46 Injector: crawlDb: TestCrawl/crawldb Injector: urlDir: urls/seed.txt Injector: Converting injected urls to crawl db entries. 2013-04-23 22:00:46.562 java[1047:1903] Unable to load realm info from SCDynamicStore Injector: total number of urls rejected by filters: 0 Injector: total number of urls injected after normalization and filtering: 1 Injector: Merging injected urls into crawl db. Injector: finished at 2013-04-23 22:01:01, elapsed: 00:00:14 2013年 4月23日 星期二 22时01分01秒 CST : Iteration 1 of 2 Generating a new segment 2013-04-23 22:01:01.888 java[1055:1903] Unable to load realm info from SCDynamicStore Generator: starting at 2013-04-23 22:01:02 Generator: Selecting best-scoring urls due for fetch. Generator: filtering: false Generator: normalizing: true Generator: topN: 50000 Generator: Partitioning selected urls for politeness. Generator: segment: TestCrawl/segments/20130423220110 Generator: finished at 2013-04-23 22:01:17, elapsed: 00:00:15 Operating on segment : drwxr-xr-xnn3ncaryannstaffnn102nn4n23n22:01n20130423220110 Fetching : drwxr-xr-xnn3ncaryannstaffnn102nn4n23n22:01n20130423220110 Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property. Fetcher: starting at 2013-04-23 22:01:18 Fetcher: segment: TestCrawl/segments/drwxr-xr-xnn3ncaryannstaffnn102nn4n23n22:01n20130423220110 Fetcher Timelimit set for : 1366736478177 2013-04-23 22:01:18.308 java[1068:1903] Unable to load realm info from SCDynamicStore Fetcher: java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: drwxr-xr-xnn3ncaryannstaffnn102nn4n23n22:01n20130423220110 at org.apache.hadoop.fs.Path.initialize(Path.java:148) at org.apache.hadoop.fs.Path.<init>(Path.java:126) at org.apache.hadoop.fs.Path.<init>(Path.java:50) at org.apache.hadoop.fs.FileSystem.globPathsLevel(FileSystem.java:1084) at org.apache.hadoop.fs.FileSystem.globPathsLevel(FileSystem.java:1087) at org.apache.hadoop.fs.FileSystem.globPathsLevel(FileSystem.java:1087) at org.apache.hadoop.fs.FileSystem.globPathsLevel(FileSystem.java:1087) at org.apache.hadoop.fs.FileSystem.globPathsLevel(FileSystem.java:1087) at org.apache.hadoop.fs.FileSystem.globPathsLevel(FileSystem.java:1087) at org.apache.hadoop.fs.FileSystem.globPathsLevel(FileSystem.java:1087) at org.apache.hadoop.fs.FileSystem.globPathsLevel(FileSystem.java:1087) at org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1023) at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:987) at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:177) at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:40) at org.apache.nutch.fetcher.Fetcher$InputFormat.getSplits(Fetcher.java:105) at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:989) at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:981) at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:174) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:897) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:824) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1261) at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1332) at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:1368) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:1341) Caused by: java.net.URISyntaxException: Relative path in absolute URI: drwxr-xr-xnn3ncaryannstaffnn102nn4n23n22:01n20130423220110 at java.net.URI.checkPath(URI.java:1788) at java.net.URI.<init>(URI.java:734) at org.apache.hadoop.fs.Path.initialize(Path.java:145) ... 30 more All I did was following the totur as follows: 1. download nutch bin from: http://mirror.esocc.com/apache/nutch/1.6/apache-nutch-1.6-bin.zip 2. unzip and step into the dir: apache-nutch-1.6 3. in my home dir i setup JAVA_HOME in .bash_profile like: JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home export JAVA_HOME 4. change the content in conf/nutch-site.xml to follows: <configuration> <property> <name>http.agent.name</name> <value>NutchSpider</value> </property> </configuration> 5. under dir: apache-nutch-1.6, excute: mkdir -p urls cd urls touch seed.txt 6. edit seed.txt with content: http://nutch.apache.org/ 7. then edit file conf/regex-urlfilter.txt and replace # accept anything else +. with +^http://([a-z0-9]*\.)*nutch.apache.org/ 8. finally, i run comand under dir :apache-nutch-1.6 as follows: MaohuaLiu-MacBook-Pro:apache-nutch-1.6 carya$ bin/crawl urls/seed.txt TestCrawl http://localhost:8983/solr/ 2 9. at the end show the error message as mentioned before. please help me to solve this problem, thanks very much. my java version: MaohuaLiu-MacBook-Pro:apache-nutch-1.6 carya$ java -version java version "1.6.0_43" Java(TM) SE Runtime Environment (build 1.6.0_43-b01-447-11M4203) Java HotSpot(TM) 64-Bit Server VM (build 20.14-b01-447, mixed mode) Max OS X version 10.7.5 Best Regards. -------------------------------------- Maohua Liu Email: [email protected]
