Re: [PROPOSAL] index server project
On 10/19/06, Steven Parkes <[EMAIL PROTECTED]> wrote: You mention partitioning of indexes, though mostly around delete. What about scalability of corpus size? Definitely in scope. Solr already has scalability of search volume via searchers behind of a load balancer all getting their index from a master. The problem comes when an index is too big to get decent latency for a single query, and that's when you need to partiton the index into "shards" to use google terminology. Would partitioning be effective for that, too? Yes, to a certain extent. At some point you run into network bandwidth issues if you go deep into rankings. What about scalability of ingest rate? As it relates to indexing, I think nutch already has that base covered. What are you thinking, in terms of size? Is this a 10 node thing? I'm personally interested in perhaps 10 to 20 index shards, with multiple replicas of each shard for HA and query load scalability. A 1000 node thing? More? Bigger is cool, but raises a lot of issues. Should be possible, but I won't personally be looking for that. I think scaling effectively will be partially in the hands of the client and how it chooses to merge results from shards. How dynamic? Can nodes come and go? Unplanned: yes. HA is personally key for me. Planned (adding capacity gracefully): it would be nice. I actually hadn't planned it for Solr. Are you going to assume homogeneity of nodes? Hardware homogeneity? That might be out of scope... I'd start off without worrying about it in any case. What about add/modify/delete to search visibility latency? Close to batch/once-a-day or real-time? Anywhere in between I'd think. "Realtime" latencies of minutes or longer are normally fine. -Yonik http://incubator.apache.org/solr Solr, the open-source Lucene search server
RE: [PROPOSAL] index server project
I like the idea. I'm trying to figure out, in broad strokes, the overarching goals. Forgive me if this is obvious, I just want to be clear. The major goal is scale, right? A distributed server provides more oomph than a single-node server can. There are a number of dimensions in scale. You mention replication of indexes, so scalability of search volume is in scope, right? You mention partitioning of indexes, though mostly around delete. What about scalability of corpus size? Would partitioning be effective for that, too? What about scalability of ingest rate? What are you thinking, in terms of size? Is this a 10 node thing? A 1000 node thing? More? Bigger is cool, but raises a lot of issues. How dynamic? Can nodes come and go? Are you going to assume homogeneity of nodes? What about add/modify/delete to search visibility latency? Close to batch/once-a-day or real-time? I think it's definitely something people want. Actually, I think we could answer these questions in different ways and for every answer, we'd find people that would want it. But they would probably be different people.
Re: [Fwd: [PROPOSAL] index server project]
I am not sure this is (somehow) related, but I think I have noticed some project on a Sun contest (it was the big prize winner). I cannot retrieve it now, but hopefully somebody else will. ./alex -- .w( the_mindstorm )p. On 10/19/06, Stefan Groschupf <[EMAIL PROTECTED]> wrote: Hi Doug, we discussed the need of such a tool several times internally and developed some workarounds for nutch, so I would be definitely interested to contribute to such a project. Having a separated project that depends on hadoop would be the best case for our usecases. Best, Stefan Am 18.10.2006 um 23:35 schrieb Doug Cutting: > FYI, I just pitched a new project you might be interested in on > [EMAIL PROTECTED] Dunno if you subscribe to that list, so I'm > spamming you. If it sounds interesting, please reply there. My > management at Y! is interested in this, so I'm 'in'. > > Doug > > Original Message > Subject: [PROPOSAL] index server project > Date: Wed, 18 Oct 2006 14:17:30 -0700 > From: Doug Cutting <[EMAIL PROTECTED]> > Reply-To: general@lucene.apache.org > To: general@lucene.apache.org > > It seems that Nutch and Solr would benefit from a shared index serving > infrastructure. Other Lucene-based projects might also benefit from > this. So perhaps we should start a new project to build such a thing. > This could start either in java/contrib, or as a separate sub-project, > depending on interest. > > Here are some quick ideas about how this might work. > > An RPC mechanism would be used to communicate between nodes (probably > Hadoop's). The system would be configured with a single master node > that keeps track of where indexes are located, and a number of slave > nodes that would maintain, search and replicate indexes. Clients > would > talk to the master to find out which indexes to search or update, then > they'll talk directly to slaves to perform searches and updates. > > Following is an outline of how this might look. > > We assume that, within an index, a file with a given name is written > only once. Index versions are sets of files, and a new version of an > index is likely to share most files with the prior version. Versions > are numbered. An index server should keep old versions of each index > for a while, not immediately removing old files. > > public class IndexVersion { > String Id; // unique name of the index > int version; // the version of the index > } > > public class IndexLocation { > IndexVersion indexVersion; > InetSocketAddress location; > } > > public interface ClientToMasterProtocol { > IndexLocation[] getSearchableIndexes(); > IndexLocation getUpdateableIndex(String id); > } > > public interface ClientToSlaveProtocol { > // normal update > void addDocument(String index, Document doc); > int[] removeDocuments(String index, Term term); > void commitVersion(String index); > > // batch update > void addIndex(String index, IndexLocation indexToAdd); > > // search > SearchResults search(IndexVersion i, Query query, Sort sort, int n); > } > > public interface SlaveToMasterProtocol { > // sends currently searchable indexes > // recieves updated indexes that we should replicate/update > public IndexLocation[] heartbeat(IndexVersion[] searchableIndexes); > } > > public interface SlaveToSlaveProtocol { > String[] getFileSet(IndexVersion indexVersion); > byte[] getFileContent(IndexVersion indexVersion, String file); > // based on experience in Hadoop, we probably wouldn't really use > // RPC to send file content, but rather HTTP. > } > > The master thus maintains the set of indexes that are available for > search, keeps track of which slave should handle changes to an > index and > initiates index synchronization between slaves. The master can be > configured to replicate indexes a specified number of times. > > The client library can cache the current set of searchable indexes and > periodically refresh it. Searches are broadcast to one index with > each > id and return merged results. The client will load-balance both > searches and updates. > > Deletions could be broadcast to all slaves. That would probably be > fast > enough. Alternately, indexes could be partitioned by a hash of each > document's unique id, permitting deletions to be routed to the > appropriate slave. > > Does this make sense? Does it sound like it would be useful to Solr? > To Nutch? To others? Who would be interested and able to work on it? > > Doug > ~~~ 101tec Inc. search tech for web 2.1 Menlo Park, California http://www.101tec.com
Re: [Fwd: [PROPOSAL] index server project]
Hi Doug, we discussed the need of such a tool several times internally and developed some workarounds for nutch, so I would be definitely interested to contribute to such a project. Having a separated project that depends on hadoop would be the best case for our usecases. Best, Stefan Am 18.10.2006 um 23:35 schrieb Doug Cutting: FYI, I just pitched a new project you might be interested in on [EMAIL PROTECTED] Dunno if you subscribe to that list, so I'm spamming you. If it sounds interesting, please reply there. My management at Y! is interested in this, so I'm 'in'. Doug Original Message Subject: [PROPOSAL] index server project Date: Wed, 18 Oct 2006 14:17:30 -0700 From: Doug Cutting <[EMAIL PROTECTED]> Reply-To: general@lucene.apache.org To: general@lucene.apache.org It seems that Nutch and Solr would benefit from a shared index serving infrastructure. Other Lucene-based projects might also benefit from this. So perhaps we should start a new project to build such a thing. This could start either in java/contrib, or as a separate sub-project, depending on interest. Here are some quick ideas about how this might work. An RPC mechanism would be used to communicate between nodes (probably Hadoop's). The system would be configured with a single master node that keeps track of where indexes are located, and a number of slave nodes that would maintain, search and replicate indexes. Clients would talk to the master to find out which indexes to search or update, then they'll talk directly to slaves to perform searches and updates. Following is an outline of how this might look. We assume that, within an index, a file with a given name is written only once. Index versions are sets of files, and a new version of an index is likely to share most files with the prior version. Versions are numbered. An index server should keep old versions of each index for a while, not immediately removing old files. public class IndexVersion { String Id; // unique name of the index int version; // the version of the index } public class IndexLocation { IndexVersion indexVersion; InetSocketAddress location; } public interface ClientToMasterProtocol { IndexLocation[] getSearchableIndexes(); IndexLocation getUpdateableIndex(String id); } public interface ClientToSlaveProtocol { // normal update void addDocument(String index, Document doc); int[] removeDocuments(String index, Term term); void commitVersion(String index); // batch update void addIndex(String index, IndexLocation indexToAdd); // search SearchResults search(IndexVersion i, Query query, Sort sort, int n); } public interface SlaveToMasterProtocol { // sends currently searchable indexes // recieves updated indexes that we should replicate/update public IndexLocation[] heartbeat(IndexVersion[] searchableIndexes); } public interface SlaveToSlaveProtocol { String[] getFileSet(IndexVersion indexVersion); byte[] getFileContent(IndexVersion indexVersion, String file); // based on experience in Hadoop, we probably wouldn't really use // RPC to send file content, but rather HTTP. } The master thus maintains the set of indexes that are available for search, keeps track of which slave should handle changes to an index and initiates index synchronization between slaves. The master can be configured to replicate indexes a specified number of times. The client library can cache the current set of searchable indexes and periodically refresh it. Searches are broadcast to one index with each id and return merged results. The client will load-balance both searches and updates. Deletions could be broadcast to all slaves. That would probably be fast enough. Alternately, indexes could be partitioned by a hash of each document's unique id, permitting deletions to be routed to the appropriate slave. Does this make sense? Does it sound like it would be useful to Solr? To Nutch? To others? Who would be interested and able to work on it? Doug ~~~ 101tec Inc. search tech for web 2.1 Menlo Park, California http://www.101tec.com
Re: OT: super fast MySQL full-text searching.
If you're interested, we use the following pattern to do incremental updates between a database and a Lucene index. 1) Add a field to the database table you wish to index called "DateUpdate". Update this date whenever a field in the table is changed. 2) Create a new database table to store the ID of any item that is deleted from the table above. I'll referrer to this as the "Deleted" table. 3) Have an indexer application that runs every X minutes, and does the following: a) Load all the items from the "Deleted" table and remove them from the Lucene index b) Load all the items from the main table with a "DateUpdate" date greater than the last time the indexer application ran. Delete these items from the Lucene index, and then reinsert them with the newer data. c) Purge all the items from the "Deleted" table. d) Save the date of the last "DateUpdated" item you processed, and use this date to load items the next time the indexer application runs. This is an oversimplification, since you need to consider failover etc, and there may be other factors that dictate your search indexing rules. But it gives you a general idea. I'd be curious to hear/discuss other solutions to this. Monsur On 10/19/06, Scott <[EMAIL PROTECTED]> wrote: I have tried Senna that is an embeddable fulltext search engine. It is embbeded into MySQL. http://qwik.jp/senna/ I inserted 1,000,000 of documents by using INSERT INTO sql, and I can search documents by using SELECT * FROM table WHERE MATCH(field_name) AGAINST('search-words'). It is based on SQL, so easy to use and support incremental update. I don't do benchmark test yet but it's not slow. I think that the Lucene need to support incremental update futurely. -- Scott
OT: super fast MySQL full-text searching.
I have tried Senna that is an embeddable fulltext search engine. It is embbeded into MySQL. http://qwik.jp/senna/ I inserted 1,000,000 of documents by using INSERT INTO sql, and I can search documents by using SELECT * FROM table WHERE MATCH(field_name) AGAINST('search-words'). It is based on SQL, so easy to use and support incremental update. I don't do benchmark test yet but it's not slow. I think that the Lucene need to support incremental update futurely. -- Scott