Hi John, I didn't run a long running reliable benchmark, so at the moment I can't really speak of numbers. Suggestions and help on performance testing are welcome: I guess it will shine in some situations, not necessarily all, so really choosing a correct ratio of concurrent writers/searches, number of nodes in the cluster and resources per node will never be fair enough to compare this Directory with others.
On paper the premises are good: it's all in-memory, until it fits: it will distribute data across nodes and overflow to disk is supported (called passivation). A permanent store can be configured, so you could set it to periodically flush incrementally to slower storages like a database, a filesystem, a cloud storage service. This makes it possible to avoid losing state even when all nodes are shut down. A RAMDirectory is AFAIK not recommended as you could hit memory limits and because it's basically a synchronized HashMap; Infinispan implements ConcurrentHashMap and doesn't need synchronization. Even if the data is replicated across nodes each node has it's own local cache, so when caches are warm and all segments fit in memory it should be, theoretically, the fastest Directory ever. The more it will read from disk, the more it will behave similarly to a FSDirectory with some buffers. As per Lucene's design, writes can happen only at one node at a time: one IndexWriter can own the lock, but IndexReaders and Searchers are not blocked, so when using this Directory it should behave exactly as if you had multiple processes sharing a local NIOFSdirectory: basically the situation is that you can't scale on writers, but you can scale near-linearly with readers adding in more power from more machines. Besides performance, the reasons to implement this was to be able to easily add or remove processing power to a service (clouds), make it easier to share indexes across nodes, and last but not least to remove single points of failure: all data is distributed and there is no such notion of Master: services will continue running fine when killing any node. I hope this peeks your interest, sorry if I couldn't provide numbers. Regards, Sanne On Sat, Nov 14, 2009 at 11:15 PM, John Wang <john.w...@gmail.com> wrote: > HI Sanne: > > Very interesting! > > What kinda performance should we expect with this, comparing to regular > FSDIrectory on local HD. > Thanks > -John > > On Sat, Nov 14, 2009 at 11:44 AM, Sanne Grinovero > <s.grinov...@sourcesense.com> wrote: >> >> Hello all, >> I'm a Lucene user and fan, I wanted to tell you that we just released >> a first technology preview of a distributed in memory Directory for >> Lucene. >> >> The release announcement: >> >> http://infinispan.blogspot.com/2009/11/second-release-candidate-for-400.html >> >> From there you'll find links to the Wiki, to the sources, to the issue >> tracker. A minimal demo is included with the sources. >> >> This was developed together with Google Summer of Code student Lukasz >> Moren and much support from the Infinispan and Hibernate Search teams, >> as we are storing the index segments on Infinispan and using it's >> atomic distributed locks to implement a Lucene LockFactory. >> >> Initial idea was to contribute it directly to Lucene, but as >> Infinispan is a LGPL dependency we had to distribute it with >> Infinispan (as the other way around would have introduced some legal >> issues); still we hope you appreciate the effort and are interested in >> giving it a try. >> All kind of feedback is welcome, especially on benchmarking >> methodologies as I yet have to do some serious performance tests. >> >> Main code, build with Maven2: >> svn co >> http://anonsvn.jboss.org/repos/infinispan/tags/4.0.0.CR2/lucene-directory/ >> infinispan-directory >> >> Demo, see the Readme: >> svn co >> http://anonsvn.jboss.org/repos/infinispan/tags/4.0.0.CR2/demos/lucene-directory/ >> lucene-demo >> >> Best Regards, >> Sanne >> >> -- >> Sanne Grinovero >> Sourcesense - making sense of Open Source: http://www.sourcesense.com >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-dev-h...@lucene.apache.org >> > > -- Sanne Grinovero Sourcesense - making sense of Open Source: http://www.sourcesense.com --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
