Zipping cores/shards is in the latest patch...
On Fri, Jan 15, 2010 at 11:22 AM, Andrzej Bialecki <a...@getopt.org> wrote: > On 2010-01-15 20:13, Ted Dunning wrote: >> >> This can also be a big performance win. Jason Venner reports significant >> index and cluster start time improvements by indexing to local disk, >> zipping >> and then uploading the resulting zip file. Hadoop has significant file >> open >> overhead so moving one zip file wins big over many index component files. >> There is a secondary bandwidth win as well. > > Indeed, this one should be easy to add to this patch. Unless Jason & Jason > already cooked a patch for this? ;) > >> >> On Fri, Jan 15, 2010 at 8:34 AM, Andrzej Bialecki >> (JIRA)<j...@apache.org>wrote: >> >>> >>> HDFS doesn't support enough POSIX to support writing Lucene indexes >>> directly to HDFS - for this reason indexes are always created on local >>> storage of each node, and then after closing they are copied to HDFS. > > > > > -- > Best regards, > Andrzej Bialecki <>< > ___. ___ ___ ___ _ _ __________________________________ > [__ || __|__/|__||\/| Information Retrieval, Semantic Web > ___|||__|| \| || | Embedded Unix, System Integration > http://www.sigram.com Contact: info at sigram dot com > >
