> I'm not sure what the problem was but I made some changes to > MetaTagsIndexer.java, adding new doc.adds.
Also I have updated the > schema.xml and nutch-site.xml so perhaps there was some mismatch between > fieldnames. > by default the indexer handles only metatag.keywords and metatag.description. In your case you probably had to add 'metatag.title' as well. Making it more generic would be doable but annoying as it requires some knowledge about the nature of the fields e.g. split into multiple fields as we do for keywords etc... Definitely easier to do like you did and modify the indexer if needs be - at least the parsing side of things is already sorted > > I had to do this to enable custom fields to be indexed. It would be great > if this could be done in configuration only. > do you mean adding the corresponding field names in schema.xml? We could do that as part of the commit for the default fields added by the indexer (keywords + description). J. > > > > Claus Daldorph Nielsen > > Theilgaard Mortensen a/s > > > > > Julien Nioche <[email protected]> > 25-05-2010 11:45 > Please respond to > [email protected] > > > To > [email protected] > cc > > Subject > Re: Parse and index meta tags in Nutch 1.0 > > > > > > > Hi Claus, > > Glad you got it to work. Do you know what the problem was? > > BTW you can vote for issues you like in Jira - if enough people find this > plugin useful I'll commit it to the trunk > > J. > > On 25 May 2010 08:57, Claus Daldorph Nielsen <[email protected]> wrote: > > > Julien, > > > > Thank you so much I really appreciate your help. I have now managed to > get > > Nutch to index meta tags in my Solr index (I am using Luke to verify > that > > the correct content is in my index). Only thing left now is to find out > > how to search and get content from the new fields in Solr. > > > > > > > > Claus Daldorph Nielsen > > > > Theilgaard Mortensen a/s > > > > > > > > Julien Nioche <[email protected]> > > 21-05-2010 17:18 > > Please respond to > > [email protected] > > > > > > To > > [email protected] > > cc > > > > Subject > > Re: Parse and index meta tags in Nutch 1.0 > > > > > > > > > > > > > > You can : > > - run *bin/nutch org.apache.nutch.parse.ParserChecker *and check that > you > > are getting metatag.* in the parse-metadata > > - check in the log that the parse-metatags is really loaded > > - run 'ant test-plugins' and see the output in build/parse-metatags > > - check that you've added the field definitions in the SOLR schema > > - index with Lucene and use Luke to check that the fields are created > > > > > > On 21 May 2010 15:54, Claus Daldorph Nielsen <[email protected]> wrote: > > > > > I never got this to work. So if anybody have some ideas for debugging > > then > > > please post your ideas. > > > > > > The problem is that the meta tags are never found or added to the Solr > > > index. I have no idea why. > > > > > > > > > > > > Claus Daldorph Nielsen > > > > > > Theilgaard Mortensen a/s > > > Niels Hemmingsens gade 9 > > > 1153 København K > > > > > > Tlf: 33448555 > > > > > > > > > > > > Julien Nioche <[email protected]> > > > 21-05-2010 13:33 > > > Please respond to > > > [email protected] > > > > > > > > > To > > > [email protected] > > > cc > > > > > > Subject > > > Re: Parse and index meta tags in Nutch 1.0 > > > > > > > > > > > > > > > > > > > > > Have you checked the discussion in > > > > > > http://lucene.472066.n3.nabble.com/description-and-keywords-td690681.html? > > > What have you modified in nutch-site.xml? > > > > > > j. > > > > > > On 21 May 2010 12:15, Claus Daldorph Nielsen <[email protected]> wrote: > > > > > > > Julien, > > > > > > > > Thanks it looks much like what I need. I have applied the patch and > > > added > > > > the lines to nutch-site.xml and then rebuild the Nutch project. But > > > still > > > > I don't see any metatags in my index. Do you have any suggestions to > > > what > > > > I might be doing wrong? Perhaps some configuration that I missed? > > > > > > > > > > > > > > > > Claus Daldorph Nielsen > > > > > > > > Theilgaard Mortensen a/s > > > > Niels Hemmingsens gade 9 > > > > 1153 København K > > > > > > > > Tlf: 33448555 > > > > > > > > > > > > > > > > Julien Nioche <[email protected]> > > > > 21-05-2010 09:39 > > > > Please respond to > > > > [email protected] > > > > > > > > > > > > To > > > > [email protected] > > > > cc > > > > > > > > Subject > > > > Re: Parse and index meta tags in Nutch 1.0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > Claus, > > > > > > > > See https://issues.apache.org/jira/browse/NUTCH-809 and a related > > > > discussion > > > > on > > > > > > > > > > http://lucene.472066.n3.nabble.com/description-and-keywords-td690681.html > > > > > > > > Julien > > > > > > > > -- > > > > DigitalPebble Ltd > > > > http://www.digitalpebble.com > > > > > > > > On 21 May 2010 08:26, Claus Daldorph Nielsen <[email protected]> wrote: > > > > > > > > > Hi, > > > > > > > > > > I am new to Nutch and trying to get Nutch to index meta tags from > > html > > > > > pages and store them for searching in Solr. The tags are on this > > form: > > > > > <meta name="TITLE" content="Some title" /> > > > > > <meta name="KEYWORDS" content="Forum, help, build, stuff" /> > > > > > > > > > > I would like to store the tags as two different fields in the > index. > > I > > > > > have tried the example explaining how to create a plugin but the > > > example > > > > > is for Nutch 0.9 and only helps me getting started. > > > > > > > > > > I think that I should look at : > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > $NUTCH_HOME/src/plugin/parse-html/src/java/org/apache/nutch/parse/html/HtmlParser.java > > > > > > > > > > and find the line: > > > > > HTMLMetaProcessor.getMetaTags(metaTags, root, base); > > > > > > > > > > But I'm not sure how to go on from here. Any help would be > > appreciated > > > > and > > > > > you are welcome to inform me if you know of an existing plugin > that > > > will > > > > > index the meta tags. > > > > > > > > > > > > > > > > > > > > Claus Daldorph Nielsen > > > > > > > > > > Theilgaard Mortensen a/s > > > > > > > > > > > > > > > > > -- > > > DigitalPebble Ltd > > > http://www.digitalpebble.com > > > > > > > > > > > > -- > > DigitalPebble Ltd > > http://www.digitalpebble.com > > > > > > > -- > DigitalPebble Ltd > http://www.digitalpebble.com > > -- DigitalPebble Ltd http://www.digitalpebble.com
