Hi Dean, Unfortunately nothing official. If you look you will see that this plugin (if eventually integrated), will combine with two other issues which all revolve roughly around the same area.
I have never used this patch or any of the others. Anyone else? On Wed, Jan 11, 2012 at 8:54 PM, Dean Del Ponte <[email protected]>wrote: > Any documentation on how to use the patch at > https://issues.apache.org/jira/browse/NUTCH-809? > > My apologies for the newbie question. > > Thanks, > > Dean Del Ponte > > On Mon, Sep 26, 2011 at 4:08 PM, Julien Nioche < > [email protected]> wrote: > > > Hi Matt, > > > > The plugin urlmeta does NOT extract the metadata from HTML pages. The > > 'meta' > > in its name means 'crawldb metadata' > > > > You need to use the patch in > > https://issues.apache.org/jira/browse/NUTCH-809 > > > > HTH > > > > Julien > > > > > > On 26 September 2011 21:18, Wilson, Matt <[email protected] > > >wrote: > > > > > Also, > > > > > > In case this helps. I removed the Keywords field from the solr schema > to > > > see if it would generate an error when the SolrIndexer runs and it does > > not. > > > This has lead me to believe that nutch is either not indexing the meta > > > content or it is not sending the update to solr when SolrIndexer runs. > > > > > > Matt Wilson > > > > > > -----Original Message----- > > > From: lewis john mcgibbney [mailto:[email protected]] > > > Sent: Monday, September 26, 2011 3:04 PM > > > To: [email protected] > > > Subject: Re: Indexing specific metadata tags with urlmeta > > > > > > Hi Matt, > > > > > > Try changing > > > > > > <field name="keywords" type="string" stored="true" indexed="true" > > > multiValued="true"/> > > > > > > to > > > > > > <field name="Keywords" type="string" stored="true" indexed="true" > > > multiValued="true"/> as per your metadata tags. > > > > > > We also have a configuration option in nutch-site.xml which you could > > check > > > out. > > > > > > <property> > > > <name>urlmeta.tags</name> > > > <value></value> > > > <description> > > > To be used in conjunction with features introduced in NUTCH-655, > which > > > allows > > > for custom metatags to be injected alongside your crawl URLs. > > Specifying > > > those > > > custom tags here will allow for their propagation into a pages > > outlinks, > > > as > > > well as allow for them to be included as part of an index. > > > Values should be comma-delimited. ("tag1,tag2,tag3") Do not pad the > > tags > > > with > > > white-space at their boundaries, if you are using anything earlier > > than > > > Hadoop-0.21. > > > </description> > > > </property> > > > > > > On Mon, Sep 26, 2011 at 7:07 PM, Wilson, Matt > > > <[email protected]>wrote: > > > > > > > I am attempting to crawl a corporate intranet site and allow it to be > > > > searched in solr. As part of the requirements I have to be able to > > index > > > > certain metadata tags as their own field in solr (for faceted > search). > > > For > > > > example, the pages being crawled contain the following meta tag: > > > > > > > > <meta id="ctl00_hdrKeywords" name="Keywords" content="Banking, > Savings, > > > > Student Loans, CDs, Certificates of Deposit, Smart Option Loan, 529 > > > Plans" > > > > /> > > > > > > > > I have updated the nutch-site.xml with the following: > > > > > > > > <property> > > > > <name>plugin.includes</name> > > > > <value>urlmeta|protocol-httpclient|... </value> > > > > </property> > > > > <property> > > > > <name>urlmeta.tags</name> > > > > <value>keywords</value> > > > > </property> > > > > > > > > I have updated the solr schema.xml with the following addition: > > > > > > > > <field name="keywords" type="string" stored="true" indexed="true" > > > > multiValued="true"/> > > > > > > > > I can see that the field has been created in Solr via the admin > > > interface. > > > > I also see that nutch is loading the urlmeta plugin and adding the > > > > indexfilters etc in the hadroop.log. The problem is that nutch does > > not > > > > appear to be indexing the keywords field. All of the pages crawled > > have > > > the > > > > tag present and I am receiving no errors in the nutch log. I am > unsure > > > as > > > > to what I am missing. This seems to be pretty straightforward; > > however, > > > I > > > > must be misunderstanding either the urlmeta plugin or missing > something > > > in > > > > the configuration. > > > > > > > > > > > > > > > > -- > > > *Lewis* > > > > > > > > > This E-Mail has been scanned for viruses. > > > > > > > > > > > -- > > * > > *Open Source Solutions for Text Engineering > > > > http://digitalpebble.blogspot.com/ > > http://www.digitalpebble.com > > > -- *Lewis*
