Hi all,
I've been working with Nutch/Solr integration for several enterprise
search projects for clients (as well as my forthcoming Solr book). I
think there are some real issues with the paradigm, and I'd like to
propose a slightly modified approach which I've had to take myself.
I think its backwards to base mapping of the NutchDocument to
SolrDocument based on the fields in the former. There are several problems:
1) this requires Solr to support all Nutch fields, which might not be
the case (like segment). That is an unreasonable requirement
2) you can map a Nutch field to at most 2 Solr fields (i.e. one via a
<field> and one via a <copy> tag because the source attribute is the Map
key and therefore you can only have one)
3) there is no support for any transformations, literals, etc, like say
for Solr data import
For example, I've built an enterprise search tool that aggregates lots
of different data sources together and uses Nutch to crawl the intranet.
The schema doesn't match everything Nutch sends. I have some literals
that need to be set and I need transformations.
My approach was to reverse the building of the SolrDocument, and
populate the doc based on the Solr destination fields as defined in
solrindex-mapping.xml, i.e., it populates the doc based on what the
target Solr wants to receive, not just what Nutch wants to send.
The map of fields in solrindex-mapping.xml is now keyed by dest, i.e.
the Solr field name, not source. That way, I can map a source to
multiple destinations if I want. I further add a mapping type attribute
(defaults to just a simple copy from Nutch to Solr) that supports
literals and (shortly) transformations.
The change is easy, works well and fits better I think with the Solr
paradigm. I've done this change in the 1.x plugin but obviously it can
easily port to 2.x.
If you see some merit to this approach, I'd can open a JIRA and submit
the changes. I also have somewhere an apache.org account (from my
openejb days) and would be happy to actually help implement it if you'd
like. I think adding in transformations would be a further benefit.
Let me know.
Thanks,
Lajos
