Since posting this I started working on HBASE-3529, the goal of which is to integrate Lucene into HBase, with an eye towards fully integrating realtime search when it's available in Lucene. RT'll give immediate consistency of HBase put's into the search index. The first challenge has been how to perform queries on index files stored in HDFS without speed degradation.
To solve that problem, I took the general notion of HDFS-347 and instead now directly obtain a single block's java.io.File and memory map it for Lucene's usage. The benchmark's show that this system is viable for Lucene queries. The code is still rough, I will be cleaning it up and making it easier for others to assemble and try on their own. There is work to be done on splitting the indexes and moving Lucene indexes (to the local data node) when HBase rebalances a region. Perhaps we can discuss issues on the dev list. Comments are welcome.
