Hi John,
On Mon, Aug 29, 2011 at 11:27 PM, John R. Brinkema < [email protected]> wrote: > Lewis, > > After shaking off the annoyance of your "RTFM Luke" answer (I had read the > tutorial several times), If this was how you interpreted my comments then I am sorry. It was not how they were meant to be. I listened to your suggestion (I do respect my elders ... especially my > 'application-elders') and I spent the weekend reading code, scanning the > javadocs files and adding logging statements. Considering how poorly Nutch > is documented (sparse, and what is documented probably refers to an old > version), it was challenging, but worthwhile. You are correct in some areas, however can you be more precise and give examples of what kind of documen tation you would appreciate? I have been gradually working towards the goal of getting some comprehensive documentation for Nutch rolled out across the board but it is a reasonably large task. If you would like to help out then please see NUTCH-881. f > What I found: > > That I deserve a kick in the head since I was only looking in the Nutch > databases for the results of urlmeta., The Nutch databases, of course, no > longer contain indexing information; the name URLMetaIndexingFilter > (indexing !!!) should have told me. > > That still did not help when I looked in the Solr Index. After a lot of > analysis and some logging statements later, I discovered that 'urlmeta' was > not being loaded. The plugin.includes statement in the tutorial is > incorrect. It is (fragment) > > ...|index-(basic|anchor|'''**urlmeta''')| ... > > > and should be > > ...|index-(basic|anchor)| urlmeta | ... > Thanks, this was the reason that I posted a quick message to user@ incase I had missed or wrongly added any information to the writing plugin page. Thanks for pointing this out I will get it changed ASAP. > The name of the plugin is 'urlmeta' not index-urlmeta. > > Once I got urlmeta loaded, the indexing almost ran correctly. I got a Solr > error complaining that a field was undefined ... the metadata fields that I > was injecting. I solved that problem by added the two fields I was > injecting to the Solr schema.xml. With that, the indexing completed with no > errors. > > I now (I think) understand how urlmeta works. I do have two questions, > however. > > 1) Now that Solr is the official indexer for Nutch, are we still supposed > to copy the Nutch schema over to Solr? The Solr schema has gotten very > complicated recently and I am concerned about losing some Solr > functionality. > Correct, there are also various JIRA issues related to this long standing area as we intend to make it an easier and less confusing integration. What I would encourage you to do (if you have time) is strip you're defacto Solr schema back to the bare bones then add functionality on top of the nutch schema as you see fit. This was you can add value to you're Solr core whilst ensuring all Nutch data remains intact. > > 2) What is the roll of solrindex-mapping.xml ? I only added my field names > to the Solr schema.xml; I made no changes to the Nutch schema.xml nor made > any changes to solrindex-mapping.xml. > Nutch schema enables us to specify the types of fields we wish to index from the documents we crawl. nutch solr-mapping.xml provides a resource for changing/altering the types of those fields from source to target e.g. Nutch to Solr indexer. In shor5t it provides us with flexibility when building an index which can then be easily traversed and easily queried for precise results. > > All, in all, an interesting and educational weekend. > excellent > > /jb > > > > On 8/25/2011 5:11 AM, lewis john mcgibbney wrote: > >> Hi JB, >> >> We have recently finished a complete plugin tutorial which fully explains >> the functionality of the urlmeta plugin on the wiki. It can be found here >> [1], could I ask you to have a thorough look at it, and the code and if >> you >> still have questions then please reinforce them. >> >> [1] >> http://wiki.apache.org/nutch/**WritingPluginExample<http://wiki.apache.org/nutch/WritingPluginExample> >> >> Thank you >> >> On Wed, Aug 24, 2011 at 9:36 PM, John R. Brinkema<brinkema@teo.** >> uscourts.gov <[email protected]> >> >>> wrote: >>> Hi all, >>> >>> I am trying use URLmeta to inject meta data into documents that I crawl >>> and >>> I am having some problems. >>> >>> First the context: Nutch 1.3 with Solr 3.2 >>> >>> My seed url files looks like: >>> http://mySite.com/Guide/index.****<http://mySite.com/Guide/index.**> >>> html\trecommended=<http://**mySite.com/Guide/index.html%** >>> 5Ctrecommended= <http://mySite.com/Guide/index.html%5Ctrecommended=>> >>> >>> "Guide"\**tkeywords="Guide,**Policy,**JBmarker" >>> >>> I put JBmarker there so I could see where the metadata got put. >>> >>> Index.html itself is a table of contents of a guide; that is, it is >>> mostly >>> a list of outlinks to parts of the overall guide. >>> >>> My nutch-site.xml includes the following properties: >>> >>> <property> >>> <name>plugin.includes</name> >>> <value>protocol-http|****urlfilter-regex|parse-(html|** >>> tika)|index-(basic|anchor|****urlmeta)|scoring-opic|** >>> urlnormalizer-(pass|regex|****basic)</value> >>> </property> >>> <property> >>> <name>urlmeta.tags</name> >>> <value>recommended,keywords</****value> >>> </property> >>> >>> I fire up nutch to crawl and all goes well. To see what nutch did, I >>> ran >>> 'readseg -dump' and looked at the results. What I found was the >>> following: >>> >>> ... other Recno's above ... >>> >>> Recno:: 56 >>> URL:: http:/mySite.com/Guide/index.****html >>> >>> CrawlDatum:: >>> Version: 7 >>> Status: 65 (signature) >>> Fetch time: Tue Aug 23 10:08:18 EDT 2011 >>> Modified time: Wed Dec 31 19:00:00 EST 1969 >>> Retries since fetch: 0 >>> Retry interval: 0 seconds (0 days) >>> Score: 1.0 >>> Signature: 5c182af41027766eccf1ea60d11277****2c >>> >>> Metadata: >>> >>> CrawlDatum:: >>> Version: 7 >>> Status: 1 (db_unfetched) >>> Fetch time: Tue Aug 23 10:08:04 EDT 2011 >>> Modified time: Wed Dec 31 19:00:00 EST 1969 >>> Retries since fetch: 0 >>> Retry interval: 2592000 seconds (30 days) >>> Score: 1.0 >>> Signature: null >>> Metadata: recommended: Guide_ngt_: 1314108489210keywords: >>> "Guide,Policy,JBmarker" >>> >>> Content:: >>> Version: -1 >>> url: >>> http://mySite.com/Guide/index.****html<http://mySite.com/Guide/index.**html> >>> <http://mySite.com/**Guide/index.html<http://mySite.com/Guide/index.html> >>> > >>> base: >>> http://mySite.com/Guide/index.****html<http://mySite.com/Guide/index.**html> >>> <http://mySite.com/**Guide/index.html<http://mySite.com/Guide/index.html> >>> > >>> >>> ... lots more content ... >>> >>> CrawlDatum:: >>> Version: 7 >>> Status: 33 (fetch_success) >>> Fetch time: Tue Aug 23 10:08:15 EDT 2011 >>> Modified time: Wed Dec 31 19:00:00 EST 1969 >>> Retries since fetch: 0 >>> Retry interval: 2592000 seconds (30 days) >>> Score: 1.0 >>> Signature: null >>> Metadata: recommended: Guide_ngt_: 1314108489210keywords: >>> "Guide,Policy,JBmarker"_pst_: success(1), lastModified=0 >>> >>> ParseData:: >>> Version: 5 >>> Status: success(1,0) >>> Title: Guide >>> Outlinks: 60 >>> outlink: toUrl: >>> http://mySite.com/Home/About.****html<http://mySite.com/Home/About.**html> >>> <http://mySite.com/Home/**About.html >>> <http://mySite.com/Home/About.html>>anchor: >>> About Me >>> outlink: toUrl: >>> http://mySite.com/Guide/****Contact_The_Guide.html<http://mySite.com/Guide/**Contact_The_Guide.html> >>> <http://**mySite.com/Guide/Contact_The_**Guide.html<http://mySite.com/Guide/Contact_The_Guide.html>>anchor: >>> Contact Me >>> ... many more outlinks ... >>> Content Metadata: nutch.content.digest=**** >>> 5c182af41027766eccf1ea60d11277****2c >>> >>> Accept-ranges=bytes Date=Tue, 23 Aug 2011 16:28:43 GMT >>> Content-Length=28798 >>> Last-Modified=Wed, 06 Apr 2011 00:15:10 GMT nutch.crawl.score=1.0 >>> _fst_=33 >>> nutch.segment.name=****20110823100811 Content-Type=text/html >>> Connection=close Server=Netscape-Enterprise/6.0 >>> Parse Metadata: CharEncodingForConversion=****windows-1252 >>> OriginalCharEncoding=windows-****1252 >>> >>> ParseText:: >>> ... lots of parsed text ... >>> >>> Recno:: 57 >>> >>> ... and so forth. >>> >>> JBmarker does not appear anywhere else, in this segment or any of the >>> others. >>> >>> When I do a solrindex, JBmarker does not appear to be anywhere. ?? >>> >>> *What I expected* >>> >>> As I understand ULRmeta (as defined by the two nutch patches), the meta >>> data that is included with the url is injected into the seed url; that >>> is >>> to say, it is as if the lines: >>> >>> <META NAME="recommended" CONTENT="Guide"> >>> <META NAME="keywords" CONTENT="Guide,Policy,****JBmarker"> >>> >>> were in the seed url content. Furthermore, it is as if those two lines >>> were in all the outlink content of the seed url. So, I expected that >>> when I >>> looked at all the CrawlDatum and ParseData of the outlinks from the seed >>> url, I would see the same meta data as in the seed CrawlDatum and >>> ParseData. >>> Which is clearly not the case. >>> >>> As for solrindex, I assume that I have some work to do to get any special >>> metadata actions moved over to solr; a special plugin of some sort. That >>> is, urlmeta does not help get the collected metadata from Nutch to Solr. >>> >>> So what is happening? Where did I go astray? Am I analyzing the Nutch >>> dumps incorrectly? >>> >>> One other side note: I assume that Luke no longer will help me debug >>> Nutch >>> since it works with Lucene indexes and Nutch no longer create such >>> beasts. >>> Are there any tools that help with viewing Nutch databases? It seems >>> that >>> Nutch takes some liberties with the data it is dumping (e.g., the meta >>> tags >>> all concatenated together without delimiters; I assume that internally, >>> the >>> meta tags are separated somehow). >>> >>> Thanks, as always. >>> >>> >>> >>> >>> >>> >>> >>> >>> >> > > -- *Lewis*
