Alan Tanaman wrote:
Currently Nutch creates a Lucene multifile index, and makes sure any
existing compound index is converted to multifile by using the
IndexWriter.setUseCompoundFile(false) method.
This is done whenever an IndexWriter is opened in the following methods:
org.apache.nutch.indexer.Indexer.getRecordWriter
org.apache.nutch.indexer.IndexSorter.sort
org.apache.nutch.indexer.IndexMerger.merge
Is there a technical constraint as to why Nutch should ensure usage of
multifile (or prevent compound) and not allow the type to be set by a
property setting?
Does anyone object to/support a patch to allow this to be configurable?
Multifile indexes are somewhat faster, and require much less temporary
space during indexing. Why would you want to use the compound format
with Nutch? The typical use of Nutch is that you work with a single or
at most couple (few) indexes per machine - in such case, regular
non-compound index works better, and there is no danger of running out
of file handles.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
