Hi all,
I am trying use URLmeta to inject meta data into documents that I crawl
and I am having some problems.
First the context: Nutch 1.3 with Solr 3.2
My seed url files looks like:
http://mySite.com/Guide/index.html\trecommended="Guide"\tkeywords="Guide,Policy,JBmarker";
I put JBmarker there so I could see where the metadata got put.
Index.html itself is a table of contents of a guide; that is, it is
mostly a list of outlinks to parts of the overall guide.
My nutch-site.xml includes the following properties:
<property>
<name>plugin.includes</name>
<value>protocol-http|urlfilter-regex|parse-(html|tika)|index-(basic|anchor|urlmeta)|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
</property>
<property>
<name>urlmeta.tags</name>
<value>recommended,keywords</value>
</property>
I fire up nutch to crawl and all goes well. To see what nutch did, I
ran 'readseg -dump' and looked at the results. What I found was the
following:
... other Recno's above ...
Recno:: 56
URL:: http:/mySite.com/Guide/index.html
CrawlDatum::
Version: 7
Status: 65 (signature)
Fetch time: Tue Aug 23 10:08:18 EDT 2011
Modified time: Wed Dec 31 19:00:00 EST 1969
Retries since fetch: 0
Retry interval: 0 seconds (0 days)
Score: 1.0
Signature: 5c182af41027766eccf1ea60d112772c
Metadata:
CrawlDatum::
Version: 7
Status: 1 (db_unfetched)
Fetch time: Tue Aug 23 10:08:04 EDT 2011
Modified time: Wed Dec 31 19:00:00 EST 1969
Retries since fetch: 0
Retry interval: 2592000 seconds (30 days)
Score: 1.0
Signature: null
Metadata: recommended: Guide_ngt_: 1314108489210keywords:
"Guide,Policy,JBmarker"
Content::
Version: -1
url: http://mySite.com/Guide/index.html
base: http://mySite.com/Guide/index.html
... lots more content ...
CrawlDatum::
Version: 7
Status: 33 (fetch_success)
Fetch time: Tue Aug 23 10:08:15 EDT 2011
Modified time: Wed Dec 31 19:00:00 EST 1969
Retries since fetch: 0
Retry interval: 2592000 seconds (30 days)
Score: 1.0
Signature: null
Metadata: recommended: Guide_ngt_: 1314108489210keywords:
"Guide,Policy,JBmarker"_pst_: success(1), lastModified=0
ParseData::
Version: 5
Status: success(1,0)
Title: Guide
Outlinks: 60
outlink: toUrl: http://mySite.com/Home/About.html anchor: About Me
outlink: toUrl: http://mySite.com/Guide/Contact_The_Guide.html
anchor: Contact Me
... many more outlinks ...
Content Metadata: nutch.content.digest=5c182af41027766eccf1ea60d112772c
Accept-ranges=bytes Date=Tue, 23 Aug 2011 16:28:43 GMT
Content-Length=28798 Last-Modified=Wed, 06 Apr 2011 00:15:10 GMT
nutch.crawl.score=1.0 _fst_=33 nutch.segment.name=20110823100811
Content-Type=text/html Connection=close Server=Netscape-Enterprise/6.0
Parse Metadata: CharEncodingForConversion=windows-1252
OriginalCharEncoding=windows-1252
ParseText::
... lots of parsed text ...
Recno:: 57
... and so forth.
JBmarker does not appear anywhere else, in this segment or any of the
others.
When I do a solrindex, JBmarker does not appear to be anywhere. ??
*What I expected*
As I understand ULRmeta (as defined by the two nutch patches), the meta
data that is included with the url is injected into the seed url; that
is to say, it is as if the lines:
<META NAME="recommended" CONTENT="Guide">
<META NAME="keywords" CONTENT="Guide,Policy,JBmarker">
were in the seed url content. Furthermore, it is as if those two lines
were in all the outlink content of the seed url. So, I expected that
when I looked at all the CrawlDatum and ParseData of the outlinks from
the seed url, I would see the same meta data as in the seed CrawlDatum
and ParseData. Which is clearly not the case.
As for solrindex, I assume that I have some work to do to get any
special metadata actions moved over to solr; a special plugin of some
sort. That is, urlmeta does not help get the collected metadata from
Nutch to Solr.
So what is happening? Where did I go astray? Am I analyzing the Nutch
dumps incorrectly?
One other side note: I assume that Luke no longer will help me debug
Nutch since it works with Lucene indexes and Nutch no longer create such
beasts. Are there any tools that help with viewing Nutch databases? It
seems that Nutch takes some liberties with the data it is dumping (e.g.,
the meta tags all concatenated together without delimiters; I assume
that internally, the meta tags are separated somehow).
Thanks, as always.
