Analysing the nutch sh script I found that the classes are actually read inside the job file, in my case the nutch-0.9.job file. Though I haven't figured out why this archive also contains the /plugin repository and all the files found in the ./conf directory, the patch did its work and I don't have the "MultiReader.java:113" error anymore.
I still have one question: I have a few pdfs files that I'd like to crawl that are images of scan documents. Is there a plugin that has someking of OCR text extraction tool? Good week-end to all of you, David -----Original Message----- From: POIRIER David [mailto:[EMAIL PROTECTED] Sent: vendredi, 16. mai 2008 12:47 To: [email protected] Subject: RE: unable to correctly fetch https pages Vijav, I did a few things the right way this time, I think. First I created an eclipse project containing all the sources. Second I downloaded the patch we were talking about and instead of applying it using the patch command, it wasn't recognized by cygwin, I simply did the changes myself in the source code. Then, using the jar Ant command I rebuilt a nutch-0.9.jar archive. I just want to confirm something with you: When using the bin/crawl command, the needed archive is the jar file, nutch-0.9.jar in my case, available in the $NUTCH_HOME directory? Again, thank you, David -----Original Message----- From: Vijay Krishnan [mailto:[EMAIL PROTECTED] Sent: vendredi, 16. mai 2008 10:51 To: [email protected] Subject: Re: unable to correctly fetch https pages I haven't checked out the latest trunk, so no idea about this error. Vijay On Fri, May 16, 2008 at 1:49 AM, POIRIER David <[EMAIL PROTECTED]> wrote: > Hello, > > Thanks Vijay. > > I downloaded the last trunk and deployed it on my Windows system (I'm > using cygwin to interact with Nutch). I took back my modified conf files > and added my custom plugins into the plugin directory. But things are > not working... When I try to lunch a crawl I get the following error: > > Injector: Converting injected urls to crawl db entries. > Exception in thread "main" java.io.IOException: Failed to get the > current user's information: Login failed: bash: > /cygdrive/c/cygwin/bin/groups: /bin/sh: bad interpreter: No such file or > directory > > at > org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:466) > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:805) > at org.apache.nutch.crawl.Injector.inject(Injector.java:157) > at org.apache.nutch.crawl.Crawl.main(Crawl.java:113) > > > Af if it was looking for /bin/sh. Any idea what this might be? > > Thank you, > > David > > > > > -----Original Message----- > From: Vijay Krishnan [mailto:[EMAIL PROTECTED] > Sent: vendredi, 16. mai 2008 01:11 > To: [email protected] > Subject: Re: unable to correctly fetch https pages > > Yes, you can get Nutch out of the trunk or apply the following patch: > > http://www.mail-archive.com/[EMAIL PROTECTED]/msg12 > 893.html > > I had the same problem and this patch handled it. > > > Vijay > > On Thu, May 15, 2008 at 9:16 AM, <[EMAIL PROTECTED]> wrote: >> I would just get Nutch out of the trunk - it's quite stable. >> >> >> Otis >> -- >> Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch >> >> >> >> >> ----- Original Message ---- >> > From: POIRIER David <[EMAIL PROTECTED]> >> > To: [email protected] >> > Sent: Thursday, May 15, 2008 11:11:28 AM >> > Subject: unable to correctly fetch https pages >> > >> > Hello, >> > >> > >> > >> > I'm trying to fetch a web site using the https protocol. I'm using > Nutch >> > version 0.9 and I activated the protocol-httpclient plugin. The > hadoop >> > logs are set in debug mode. >> > >> > >> > >> > When checking the logs I can see that my seed url seems to be > fetch: >> > >> > "2008-05-15 12:19:04,341 INFO fetcher.Fetcher - fetching >> > https://www.aWebSite.xyz/aPage.htm"; >> > >> > >> > >> > But none of the links on this page are actually found and the > process >> > finally crash with the following error: >> > >> > >> > >> > java.lang.ArrayIndexOutOfBoundsException: -1 >> > >> > at >> > org.apache.lucene.index.MultiReader.isDeleted(MultiReader.java:113) >> > >> > at >> > > org.apache.nutch.indexer.DeleteDuplicates$InputFormat$DDRecordReader.nex >> > t(DeleteDuplicates.java:176) >> > >> > at > org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:157) >> > >> > at > org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:46) >> > >> > at > org.apache.hadoop.mapred.MapTask.run(MapTask.java:175) >> > >> > at >> > > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:126) >> > >> > >> > >> > I am 100% certain that the links on the seed page are not excluded >> > because of the regex rules used by the urlfilter-regex plugin. I > tried >> > using the urlfilter-prefix and the urlfilter-suffix plugin, I > wasn't >> > luckier. >> > >> > >> > >> > I found that a bug (NUTCH-593) generating the same error was fixed > by >> > Andrzej Bialecki in February, could this fix help me? What is the >> > easiest way for me to get this fix without actually using the > complete, >> > and I guess still unstable, version 1.0 of Nutch? >> > >> > >> > >> > Thanks, >> > >> > >> > >> > David >> >> > > > > -- > Vijay Krishnan > Founder, Infoaxe Inc. > http://www.cs.stanford.edu/~vijayk > http://www.infoaxe.com/hiring.html > -- Vijay Krishnan Founder, Infoaxe Inc. http://www.cs.stanford.edu/~vijayk http://www.infoaxe.com/hiring.html
