[
https://issues.apache.org/jira/browse/NUTCH-520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doğacan Güney updated NUTCH-520:
--------------------------------
Attachment: RFC_multiple_index_backends.patch
Here is my proposal on how we can do it along with a patch:
i) Add a NutchDocument class:
A NutchDocument contains a mapping from String-s to List<String>-s as
fields, a metadata (to be explained later) and score. NutchDocument fields
doesn't contain any information about how it is meant to be indexed or stored
(not entirely true, explained later). These options are missing because
different backends may not represent the same options. For example, solr
doesn't (AFAIK) allow you to change how a field is stored at runtime. Also, one
may want to index to a MySQL database (I don't know why, but it is possible),
which again doesn't provide storage or indexing options.
ii) Add a NutchIndexWriter interface:
NutchIndexWriter is the interface to be implemented if you want to add
another indexing backend to nutch. A NutchIndexWriter writes,
not-so-surprisingly, NutchDocument-s. Implementations are meant to take the
NutchDocument, convert it into their internal format and then write the
converted data. Also, Indexer.OutputFormat creates a list of enabled classes
that implement NutchIndexWriter. Upon each call to
OutputFormat.RecordWriter.write(Text url, NutchDocument document), this method
delegates it to NutchIndexWriter.write(NutchDocument doc).
This patch adds two classes that implement NutchIndexWriter. LuceneWriter
and SolrWriter.
iii) Allow indexing filters to define index-level and document-level metadata:
NutchDocument fields are simple key/value pairs and LuceneWriter can't
determine how to store/index them by just looking at the fields. Because of
this, two types of metadata are defined:
1) Index-level: Index level metadata is meant to be valid for all documents
and doesn't change for every document. Metadata-s are free form string,string[]
pairs that are meant to be picked up by underlying index backends.
A new method "addIndexMeta" is added to ScoringFilter. This is used by
indexing filters to add 'hints' to index backends.
For example, index-basic plugin adds <"lucene.field.title",
"lucene.store.yes"> and <"lucene.field.title", "lucene.index.tokenized"> pairs,
hinting to lucene backend that field "title" is meant to be stored and
tokenized. Implementations should always prefix meta keys with the name of the
backend to avoid conflict. If lucene backend is not active, this information is
simply ignored.
2) Document-level: Per-document free form string,string[] pairs. For
example, if you normally want to store field "foo" in a lucene index, but you
do not want to do it for a specific document, you can add a
<"lucene.field.foo", "lucene.store.no"> pair to that document's metadata and
LuceneWriter will not store field value of "foo" for that document.
Extra notes:
* This patch is a very early draft. I am sure that a lot of stuff doesn't work.
However, I tested indexing a 30000 url segment to both solr and lucene and
didn't run into any problems. When only indexing to lucene, there is no
noticable performance difference from earlier nutch versions.
* NutchDocument has a add(Field) method for easy-upgrade of older indexing
filters. However, it is slower compared and should only be used for upgrading.
* I believe that this is a very important feature for nutch. (I don't know why
I am writing this as a note)
Comments, suggestions, reviews and other feedback are welcome.
> A common infrastructure for different index backends
> ----------------------------------------------------
>
> Key: NUTCH-520
> URL: https://issues.apache.org/jira/browse/NUTCH-520
> Project: Nutch
> Issue Type: Improvement
> Components: indexer
> Reporter: Doğacan Güney
> Attachments: RFC_multiple_index_backends.patch
>
>
> With the discussion of solr as a possible index and search backend, I think
> we need a new indexing architecture (that doesn't depend on lucene) that can
> use multiple backends to index.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Nutch-developers mailing list
Nutch-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nutch-developers
