In nutch-site.xml plugin.includes my custom filter is last and I have no htmlparsefilter.order so my filter should be applied last, right ?
On Wed, Jun 26, 2013 at 5:00 PM, Amit Sela <[email protected]> wrote: > So I managed to create and deploy my plugin, which initially used > content.getContent() and it worked. > Then, I wanted to parse the fetched content as DocumentFragment (by > iterating over the child nodes). > This doesn't work. I logged DocumentFragment.toString() in my > MyCustomHtmlParseFilter in filter method, and in the Parse MapReduce logs I > see: [#document-fragment: null] for all URLS. > > How do I get nutch to pass the parsed html as DocumentFragment ? Should I > state htmlparsefilter.order in nutch-site.xml ? if so, in what order ? > > Thanks. > > > > > On Tue, Jun 25, 2013 at 5:38 PM, Amit Sela <[email protected]> wrote: > >> Thanks for the prompt answer! >> >> >> On Tue, Jun 25, 2013 at 5:35 PM, Markus Jelsma < >> [email protected]> wrote: >> >>> Hi, >>> >>> Do i understand you correctly if you want all iframe src attributes on a >>> given page stored in the iframe field? >>> >>> The src attributes are not extracted and there is no facility to do so >>> right now. You should create your own HTMLParseFilter, loop through the >>> document looking for iframe tags and collect the src attribute. Then add >>> those as parse metadata. You can then index them with the index-metadata >>> plugin. I'm not sure it supports multi valued metafields in Nutch 1.6, it >>> sure will in 1.7. >>> >>> Use the bin/nutch parsechecker and indexchecker tools to check if your >>> plugin works. >>> >>> Cheers >>> >>> >>> >>> -----Original message----- >>> > From:Amit Sela <[email protected]> >>> > Sent: Tuesday 25th June 2013 16:26 >>> > To: [email protected] >>> > Subject: Fetch iframe from HTML (if exists) >>> > >>> > Hi all, >>> > >>> > I'm using nutch 1.6 with Solr 3.6.2 and I would like to index the >>> iframe >>> > src field into Solr. >>> > i.e., >>> > <iframe src="something" scrolling="" frameborder="".......> >>> > So i want to fetch the iframe and index it as iframe so that I could >>> find >>> > URLS by iframe src. >>> > >>> > I'm crawling with no depth over a seed list, and I don't want to crawl >>> to >>> > the iframe src, just to index and store it. >>> > >>> > I tried adding >>> > <name>urlmeta.tags</name> <value>iframe</value> to nutch-site.xml >>> > >>> > and >>> > <field name="iframe" type="text_general" stored="true" indexed="true" >>> > multiValued="true"/> to schema.xml >>> > >>> > and >>> > <field dest="iframe" source="iframe"/> to solrindex-mapping.xml. >>> > >>> > What am I missing ? >>> > >>> > Thanks, >>> > >>> > Amit. >>> > >>> >> >> >
