Hi, On Tue, May 20, 2008 at 5:57 AM, Rochelle Rees < [EMAIL PROTECTED]> wrote:
> Hi there, > > I have a problem with my crawl failing at: > > Dedup adding indexes in: crawls/test/indexes > Exception in thread "main" java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604) > at > org.apache.nutch.indexer.DeleteDuplicates.dedup(DeleteDuplicates.java:43 > 9) > at org.apache.nutch.crawl.Crawl.main(Crawl.java:135) This is a generic error that just tells us that job has failed. You should have a more detailed log somewhere. For example, if you have a distributed setup, check your tasktracker log files. > > > I have tried searching for threads with a similar problem and found a > number - however the only solution I could find was to install the > patches from: > https://issues.apache.org/jira/browse/NUTCH-525 > However running deleteDups.patch and RededupUnitTest.patch made no > difference whatsoever. > > Now, interestingly, my crawl runs fine on www.lovepigs.org.nz and > www.tegelchicken.co.nz, but fails when I try intranet.canterbury.ac.nz. > > Intranet.canterbury.ac.nz requires authentication, so I ran the > NUTCH-559v0.5.patch file - however the error I have occurs with or > without this patch, and regardless of what I put in the > conf/httpclient-auth.xml file. > > Does anyone have any ideas what I can do to fix this issue? > > For reference, my conf/nutch-site.xml, conf/crawl-urlfilter.txt and > urls/urls.txt files are pasted below. > > Please let me know if you need any further info. > > -------------------------------------------- > conf/nutch-site.xml > -------------------------------------------- > <?xml version="1.0"?> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > > <!-- Put site-specific property overrides in this file. --> > > <configuration> > > <property> > > <name>http.agent.name</name> > > <value>University of Canterbury Intranet</value> > > <description> > University of Canterbury Intranet > </description> > > </property> > > > > <property> > > <name>http.agent.description</name> > > <value>Intranet for University of Canterbury</value> > > <description> Intranet for University of Canterbury > > </description> > > </property> > > > > <property> > > <name>http.agent.url</name> > > <value></value> > > <description> > > </description> > > </property> > > > > <property> > > <name>http.agent.email</name> > > <value>Web Support Email</value> > > <description>[EMAIL PROTECTED] > > </description> > > </property> > > </configuration> > -------------------------------------------- > -------------------------------------------- > > conf/crawl-urlfilter.txt > -------------------------------------------- > # The url filter file used by the crawl command. > > # Better for intranet crawling. > # Be sure to change MY.DOMAIN.NAME to your domain name. > > # Each non-comment, non-blank line contains a regular expression > # prefixed by '+' or '-'. The first matching pattern in the file > # determines whether a URL is included or ignored. If no pattern > # matches, the URL is ignored. > > # skip file:, ftp:, & mailto: urls > -^(file|ftp|mailto): > > # skip image and other suffixes we can't yet parse > -\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|r > pm|tgz|mov|MOV|exe|jpeg|JPEG|bmp|BMP)$ > > # skip URLs containing certain characters as probable queries, etc. > [EMAIL PROTECTED] > > # skip URLs with slash-delimited segment that repeats 3+ times, to break > loops > -.*(/.+?)/.*?\1/.*?\1/ > > # accept hosts in MY.DOMAIN.NAME > +^http://([a-z0-9]*\.)*intranet.canterbury.ac.nz/ > > # skip everything else > -. > -------------------------------------------- > -------------------------------------------- > > > urls/urls.txt > -------------------------------------------- > http://intranet.canterbury.ac.nz > > -------------------------------------------- > -------------------------------------------- > > Regards > Rochelle Rees > Web Team, Student Recruitment and Development (SRD) > University of Canterbury, Te Whare Wananga o Waitaha > Rm: 419, Law Building > +64-3-364 2987 Ext: 6125 > [EMAIL PROTECTED] > http://www.canterbury.ac.nz/ > > For all web enquiries please contact: > [EMAIL PROTECTED] Ext: 3100 > http://www.canterbury.ac.nz/web > > -- Doğacan Güney
