I'm looking forward some strategy to build a large Lucene distributed index, but after read a lot of mail threads, it seems that Lucene hasn't a 'default solution' for that yet.
The search engine that I'm building has a 400 GB text-database (right now, growing every day and without document deleting operation) and thousands queries per day (a very small time of response is needed). I expected to use a commodity computers cluster as hardware infrastructure and the distributed strategy must allow concurrency/parallel processing/searching with all processors. Searching on Lucene site, I thought Hadoop was the right choice. After read some mail threads about this (at java lucene and hadoop, dev and user), it seems that Hadoop isn't that good (let's put this way) to deal with distributed indexes for a search engine. So, I tried Solr and read about FederatedSearch and CollectionDistribution. An 'all-machines-have-complete-index' strategy (using rsync) can improve system throughput and concurrency by each station processing different queries, but each query will spend the same amount of time that a single-node system (what sucks). When each of a N-station cluster indexing 1/N of text collection, each will machine spend less time processing queries, but all machines must process the same query at the same time (a 'goodbye, concurrency', IMO), then merge results. Neither of them looks good for me. The 'Multiple Masters' (described on FederatedSearch) solution looks good. "There could be a master for each slice of the index. An external module could provide the update interface and forward the request to the correct master based on the unique key field". That's great, has high scalability, high concurrency, smaller index for each node... I read some papers with very similar solution [1][2][3] and discussing how to solve some of this strategy issues. Did I get anything wrong (about Hadoop and Solr)? Is Multiple Masters/FederatedSearch under development? What status? Or did I should develop it for myself? Thanks for any help. Daniel [1] B. Ribeiro-Neto, J. Kitajima, G. Navarro e N. Ziviani. Parallel Generation of Inverted Files for Distributed Text Collections. In Proceedings of the XVIII international Conference of the Chilean Computer Science Society. IEEE Computer Society Washington, 1998. [2] C. Badue, R. Baeza-Yates, B. Ribeiro-Neto e N. Ziviani. Distributed Query Processing Using Partitioned Inverted Files. Eighth Symposium on String Processing and Information Retrieval (SPIRE'01), 2001. [3] C. Badue, R. Barbosa, B. Ribeiro-Neto e N. Ziviani. Basic issues on the processing of web queries. In Proceedings of the 28th Annual international ACM SIGIR Conference on Research and Development in information Retrieval, 2005.
