On 2010-01-15 20:13, Ted Dunning wrote:
This can also be a big performance win. Jason Venner reports significant
index and cluster start time improvements by indexing to local disk, zipping
and then uploading the resulting zip file. Hadoop has significant file open
overhead so moving one zip file wins big over many index component files.
There is a secondary bandwidth win as well.
Indeed, this one should be easy to add to this patch. Unless Jason &
Jason already cooked a patch for this? ;)
On Fri, Jan 15, 2010 at 8:34 AM, Andrzej Bialecki (JIRA)<j...@apache.org>wrote:
HDFS doesn't support enough POSIX to support writing Lucene indexes
directly to HDFS - for this reason indexes are always created on local
storage of each node, and then after closing they are copied to HDFS.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
