malcolm smith wrote:
I am looking to create a parser for a groupware product that would read
pages message board type web site. (Think phpBB). But rather than creating
a single Content item which is parsed and indexed to a single lucene
document, I am planning to have the parser create a master document (for the
original post) and an additional document for each reply item.
I've reviewed the code for protocol plugins, parser plugins and indexing
plugins but each interface allows for a single document or content object to
be passed around.
Am I missing something simple?
My best bet at the moment is to implement some kind of new "fake" protocol
for the reply items then I would use the http client plugin for the first
request to the page and generate outlines on the
"fakereplyto://originalurl/reply1" "fakereplyto://originalurl/reply2" to go
back through and fetch the sub page content. But this seems round-about and
would probably generate an http request for each reply on the original
page. But perhaps there is a way to lookup the original page in the segment
db before requesting it again.
Needless to say it would seem more straightforward to tackle this in some
kind of parser plugin that could break the original page into pieces that
are treated as standalone pages for indexing purposes.
Last but not least conceptually a plugin for the indexer might be able to
take a set of custom meta data for a replies "collection" and index it as
separate lucene documents - but I can't find a way to do this given the
interfaces in the indexer plugins.
Thanks in advance
Malcolm Smith
What version of Nutch are you using? This should be already possible to
do using the 1.0 release or a nightly build. ParseResult (which is what
parsers produce) can hold multiple Parse objects, each with its own URL.
The common approach to handle whole-part relationships (like zip/tar
archives, RSS, and other compound docs) is to split them in the parser
and parse each part, then give each sub-document its own URL (e.g
file.tar!myfile.txt) and add the original URL in the metadata, to keep
track of the parent URL. The rest should be handled automatically,
although there are some other complications that need to be handled as
well (e.g. don't recrawl sub-documents).
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
