Glad you managed to get it to work. I don't know what Chris meant by that, can;t see why we'd open a JIRA when we are already using the latest version
Julien On 20 July 2011 08:19, Fernando Arreola <[email protected]> wrote: > Hi, > > Nutch 1.3 currently has Tika 0.9 which is the latest official version. I > was > trying to replace the Tika in Nutch 1.3 with a Tika project which I had > modifed (Tika 0.9 with a new parser I had created). Is it still recommended > that I create a JIRA issue if it currently has the latest official version? > > Thanks, > Fernando > > On Tue, Jul 19, 2011 at 9:41 PM, Mattmann, Chris A (388J) < > [email protected]> wrote: > > > Hey Fernando, > > > > Would be great to get a JIRA issue and patch to bring > > Nutch 1.4-branch up to date with the latest Tika > > based on your experience. > > > > Thanks for your help! > > > > Cheers, > > Chris > > > > On Jul 19, 2011, at 4:48 PM, Fernando Arreola wrote: > > > > > Hi, > > > > > > You were right, it is enough to provide the right clues in the > > > tika-mimetypes.xml file. Once the correct clues got in there, thanks to > a > > > Tika developer, all I had to do was replace the jar files with mine. It > > is > > > working just as I want it now. > > > > > > Thanks everyone for the help. > > > > > > Fernando > > > > > > On Wed, Jul 13, 2011 at 1:48 AM, Julien Nioche < > > > [email protected]> wrote: > > > > > >> You probably need to make sure that conf/tika-mimetypes.xml is the > > version > > >> you've modified and contains the clues for detecting afm files. > > >> BTW out of curiosity why did you have to modify tika-core.jar? Isn't > it > > >> enough to provide the clues in tika-mimetypes.xml? > > >> > > >> Jul > > >> > > >> On 13 July 2011 01:16, Fernando Arreola <[email protected]> wrote: > > >> > > >>> Thanks, I really appreciate all the help. I used the ParserChecker > and > > I > > >>> could see the metadata my parser extracted! > > >>> > > >>> I have one more question though, I could only see the metadata my > > parser > > >>> extracted if I used the -forceAs mimetype option. Otherwise it is > > >> detected > > >>> as a text/plain file and my parser is then not called. I ran into a > > >> similar > > >>> problem in tika and added some functionality there so that Tika's > > >> detection > > >>> mechanism would not think afm files are text/plain. Does this mean > not > > >> all > > >>> of my tika changes made it in (I updated both the tika-core.jar and > > >>> tika-parsers.jar files) or does Nutch have its own file type > detection > > >>> mechanism? > > >>> > > >>> Thanks, > > >>> Fernando > > >>> > > >>> On Tue, Jul 12, 2011 at 4:54 PM, Markus Jelsma > > >>> <[email protected]>wrote: > > >>> > > >>>> > > >>>>> Thanks for the help. I seem to be getting close to what I need to > do, > > >>> but > > >>>>> not quite there. > > >>>>> > > >>>>> I downloaded Nutch 1.3 and built it on a unix machine. It built and > > >> ran > > >>>>> fine (before changing any jar files) when I tested it on the site > > >> with > > >>>> the > > >>>>> .afm files that I want to get parsed. > > >>>>> > > >>>>> I then changed the tika-core.jar, tika-parsers.jar, nutch-site.xml > > >> (to > > >>>>> enable the parse-tika plugin) and tika-mimetypes.xml files with my > > >>>> updated > > >>>>> versions. I rebuilt (no errors) and then ran the crawl command on > the > > >>>> same > > >>>>> site. The fetch seemed to work, I did not see any errors when > running > > >>> or > > >>>> in > > >>>>> the log file. There is a parse error but it is related to a pdf I > > >> have > > >>>>> linked in the site I crawled and since I am not interested in the > pdf > > >> I > > >>>>> don't think it matters. > > >>>>> > > >>>>> Now here is my completely newb question: how can I tell if the afm > > >>> files > > >>>>> were parsed correctly in the absence of errors? > > >>>> > > >>>> The ParserChecker is what you're looking for. It's a handy tool you > > can > > >>>> locally use to find out if all goes well. > > >>>> > > >>>> bin/nutch org.apache.nutch.parse.ParserChecker > > >>>> > > >>>>> > > >>>>> I looked at the files in the segments/*/parse_data directory (since > > >>> that > > >>>> is > > >>>>> where the tutorial says the metadata goes and the parser I created > > >>> mostly > > >>>>> extracts metadata) but the files aren't really readable. I also > > >> figured > > >>>>> maybe I could search for some terms I expect parser to extract but > > >>>> couldn't > > >>>>> perform a search. When I typed the following command in the > > >>> runtime/local > > >>>>> directory: > > >>>>> > > >>>>> bin/nutch org.apache.nutch.searcher.NutchBean *search_term* > > >>>>> > > >>>>> I get the following error: > > >>>>> > > >>>>> Exception in thread "main" java.lang.NoClassDefFoundError: > > >>>>> org/apache/nutch/searcher/NutchBean > > >>>>> > > >>>>> I looked in the src directory and did not find the searcher (it was > > >> in > > >>>>> there in the 1.2 version). I tried downloading both the binary and > > >> the > > >>>> src > > >>>>> distributions for 1.3 and it was in neither. Is there a different > way > > >>> to > > >>>>> perform a search in 1.3 or is there a different way I can see > > >> readable > > >>>>> results of the parsed information? > > >>>> > > >>>> There is no searcher in 1.3. It is deprecated and removed. Use Solr > > for > > >>>> indexing to confirm or use ParserChecker or the new 1.4-dev > > >>>> o.a.n.indexer.IndexingFiltersChecker. > > >>>> > > >>>>> > > >>>>> Thanks, > > >>>>> Fernando > > >>>>> > > >>>>> On Tue, Jul 12, 2011 at 11:00 AM, lewis john mcgibbney < > > >>>>> > > >>>>> [email protected]> wrote: > > >>>>>> OK so at least we seem to have sorted out the first of you're > > >>>> problems... > > >>>>>> but now face the dreaded Windows Cygwin partnership. > > >>>>>> > > >>>>>> We do not currently have an up-to-date tutorial for this. We do > > >>> however > > >>>>>> have > > >>>>>> a tutorial for older versions of Nutch which you can find here [1] > > >>> [2] > > >>>>>> > > >>>>>> I'm going to be brutally honest with you here, working with Cygwin > > >>> was > > >>>>>> horrible from my own experience. There seems to be so much > overhead > > >>> and > > >>>>>> working with almost any other OS was a significantly easier > option. > > >> I > > >>>>>> understand that this may mean a fundamental shift in you're > > >> computing > > >>>>>> style but the benefit is well worth it. > > >>>>>> > > >>>>>> [1] http://wiki.apache.org/nutch/GettingNutchRunningOnCygwin > > >>>>>> [2] > > >>>>>> > > >>>>>> > > >>>> > > >>> > > >> > > > http://wiki.apache.org/nutch/GettingNutchRunningWithWindows?highlight=%28 > > >>>>>> cygwin%29 > > >>>>>> > > >>>>>> On Tue, Jul 12, 2011 at 6:23 PM, Fernando Arreola < > > >>> [email protected] > > >>>>>> > > >>>>>>> wrote: > > >>>>>>> Hello, > > >>>>>>> > > >>>>>>> Thanks for the replies. > > >>>>>>> > > >>>>>>> I have started trying to use Nutch 1.3 after your suggestions, > > >>>>>>> especially since I am using Tika 0.9, but I am not getting > > >> anywhere > > >>>>>>> with it. I am > > >>>>>> > > >>>>>> able > > >>>>>> > > >>>>>>> to build fine but whenever I try to run any command it gives the > > >>>> error > > >>>>>>> stating that it cannot find C:\Program. For example, if I try to > > >>> run > > >>>>>>> the following command to crawl: > > >>>>>>> > > >>>>>>> runtime/local/bin/nutch crawl urls -dir crawl -depth 3 -topN 50 > > >>>>>>> > > >>>>>>> It then gives me the following error right away before any other > > >>>>>>> output: > > >>>>>>> > > >>>>>>> runtime/local/bin/nutch: line 251: exec: C:\Program: not found > > >>>>>>> > > >>>>>>> I am running on Cygwin on Windows 7, if that helps. > > >>>>>>> > > >>>>>>> As for Tika, I did modify the CompositeDetector.java file in > > >>>> tika-core > > >>>>>>> since > > >>>>>>> I added a Detector to detect the AFM files and had to make a > > >> slight > > >>>>>> > > >>>>>> change > > >>>>>> > > >>>>>>> to the CompositeDetector. I did rebuild Nutch after I changed the > > >>>> jars > > >>>>>> > > >>>>>> and > > >>>>>> > > >>>>>>> it built fine but that is when I started getting the fetch failed > > >>>>>>> error. > > >>>>>>> > > >>>>>>> Thanks, > > >>>>>>> Fernando > > >>>>>>> > > >>>>>>> On Tue, Jul 12, 2011 at 2:13 AM, Julien Nioche < > > >>>>>>> > > >>>>>>> [email protected]> wrote: > > >>>>>>>> Hi Fernando > > >>>>>>>> > > >>>>>>>>> I have made some additions (a new parser) to the Apache Tika > > >>>>>>> > > >>>>>>> application > > >>>>>>> > > >>>>>>>>> and > > >>>>>>>>> I am trying to see if I can run my new changes using the > > >> crawl > > >>>>>>> > > >>>>>>> mechanism > > >>>>>>> > > >>>>>>>> in > > >>>>>>>> > > >>>>>>>>> Nutch, but I am having some trouble updating Nutch with my > > >>>> modified > > >>>>>>> > > >>>>>>> Tika > > >>>>>>> > > >>>>>>>>> application. > > >>>>>>>>> > > >>>>>>>>> The Tika updates I made run fine if I run Tika as a > > >> standalone > > >>>>>>>>> using > > >>>>>>>> > > >>>>>>>> either > > >>>>>>>> > > >>>>>>>>> the command line or the Tika GUI. > > >>>>>>>> > > >>>>>>>> OK > > >>>>>>>> > > >>>>>>>>> I am using Nutch 1.2, 1.3 seems to not be able to run for me > > >> (I > > >>>> get > > >>>>>> > > >>>>>> an > > >>>>>> > > >>>>>>>>> error > > >>>>>>>>> saying C:/Program not found whenever I try to do anything but > > >>> 1.2 > > >>>>>>> > > >>>>>>> should > > >>>>>>> > > >>>>>>>> be > > >>>>>>>> > > >>>>>>>>> fine for what I am trying to do which is just to see the > > >> parse > > >>>>>> > > >>>>>> results > > >>>>>> > > >>>>>>>> from > > >>>>>>>> > > >>>>>>>>> the new parser I added to Tika). > > >>>>>>>>> > > >>>>>>>>> I have replaced the tika-core.jar, tika-parsers.jar and > > >>>>>>>> > > >>>>>>>> tika-mimetypes.xml > > >>>>>>>> > > >>>>>>>>> files with my versions of those files as described in the > > >>>> following > > >>>>>>> > > >>>>>>> link: > > >>>>>>>>> http://issues.apache.org/jira/browse/NUTCH-766. I also > > >> updated > > >>>> the > > >>>>>>>>> nutch-site.xml to enable the parse-tika plugin. I also > > >> updated > > >>>> the > > >>>>>>>>> parse-plugins.xml file with the following (afm files are what > > >> I > > >>>> am > > >>>>>>> > > >>>>>>> trying > > >>>>>>> > > >>>>>>>>> to > > >>>>>>>>> > > >>>>>>>>> parse): > > >>>>>>>>> <mimeType name="application/x-font-afm"> > > >>>>>>>>> > > >>>>>>>>> <plugin id="parse-tika" /> > > >>>>>>>>> > > >>>>>>>>> </mimeType> > > >>>>>>>> > > >>>>>>>> This is not necessary as by default parse-tika is used for any > > >>>>>> > > >>>>>> mime-type > > >>>>>> > > >>>>>>>> unless the mapping mime-type / parser is specified in > > >>>>>> > > >>>>>> parse-plugins.xml. > > >>>>>> > > >>>>>>>> This should not have an impact though > > >>>>>>>> > > >>>>>>>>> I am crawling a personal site in which I have links to .afm > > >>>> files. > > >>>>>>>>> If > > >>>>>> > > >>>>>> I > > >>>>>> > > >>>>>>>>> crawl before making any updates to Nutch, it fetches the > > >> files > > >>>>>>>>> fine. > > >>>>>>>> > > >>>>>>>> After > > >>>>>>>> > > >>>>>>>>> making the updates detailed above, I get the following error: > > >>>>>>>>> "fetch > > >>>>>> > > >>>>>> of > > >>>>>> > > >>>>>>>>> http://scf.usc.edu/~jfarreol/woor2___.AFM failed with: > > >>>>>> > > >>>>>>>>> java.lang.NoClassDefFoundError: > > >>>>>> org/apache/james/mime4j/MimeException". > > >>>>>> > > >>>>>>>>> Not really sure, what the issue is but my guess is that I > > >> have > > >>>> not > > >>>>>>>> > > >>>>>>>> updated > > >>>>>>>> > > >>>>>>>>> all the necessary files. Any help would be greatly > > >> appreciated. > > >>>>>>>> > > >>>>>>>> yep, sounds like you have a few jars missing. Nutch-1.2 came > > >> with > > >>>>>>> > > >>>>>>> tika-0.7, > > >>>>>>> > > >>>>>>>> which version of tika are you trying to use? > > >>>>>>>> if you just added a new parser then it would be easier to ship > > >> it > > >>>> as > > >>>>>>>> a separate jar file. I assume that you did not have to modify > > >>>>>>>> anything in tika-core, so you could use the standard tika libs > > >>> and > > >>>>>>>> simply add yours using Ivy. > > >>>>>>>> > > >>>>>>>> Nutch-1.3 (and 1.4 in SVN) contain a lot of improvements over > > >> 1.2 > > >>>> so > > >>>>>>>> it would be worth getting to the bottom of the issue you're > > >>>>>>>> encountering > > >>>>>> > > >>>>>> and > > >>>>>> > > >>>>>>>> get 1.3 to work. Moreover I am not sure that you can use a > > >>> version > > >>>> of > > >>>>>>> > > >>>>>>> Tika > > >>>>>>> > > >>>>>>>> 0.7 on Nutch 1.2 without changing parts of the code (to be > > >>> checked > > >>>>>>> > > >>>>>>> though) > > >>>>>>> > > >>>>>>>> Julien > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> -- > > >>>>>>>> * > > >>>>>>>> *Open Source Solutions for Text Engineering > > >>>>>>>> > > >>>>>>>> http://digitalpebble.blogspot.com/ > > >>>>>>>> http://www.digitalpebble.com > > >>>>>> > > >>>>>> -- > > >>>>>> *Lewis* > > >>>> > > >>> > > >> > > >> > > >> > > >> -- > > >> * > > >> *Open Source Solutions for Text Engineering > > >> > > >> http://digitalpebble.blogspot.com/ > > >> http://www.digitalpebble.com > > >> > > > > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > Chris Mattmann, Ph.D. > > Senior Computer Scientist > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > > Office: 171-266B, Mailstop: 171-246 > > Email: [email protected] > > WWW: http://sunset.usc.edu/~mattmann/ > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > Adjunct Assistant Professor, Computer Science Department > > University of Southern California, Los Angeles, CA 90089 USA > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > > -- * *Open Source Solutions for Text Engineering http://digitalpebble.blogspot.com/ http://www.digitalpebble.com
