Re: How can I create about 100000 independent indexes in Solr?
Hello Monton, I wanted to make sure that you understood me well : I really don't how well does solr scale if the number of fields increases... What I mean here is that the more distinct fields you index, the more memory you will need. So if in your schema, you have something like 15 fields declared, then storing data for 100 distinct customers would generate 1500 fields in the index. I really don't know how well would that scale. The simplest solution is one core per customer but the same issue (memory consumption) will rise at some point, I guess. There must be a clever way to do that... -- Tanguy 2012/9/26 韦震宇 weizhe...@win-trust.com Hi, Tanguy I would do as your suggestion. Best Regards! Monton - Original Message - From: Tanguy Moal tanguy.m...@gmail.com To: solr-user@lucene.apache.org; t...@statsbiblioteket.dk Sent: Tuesday, September 25, 2012 11:05 PM Subject: Re: How can I create about 10 independent indexes in Solr? That is an interesting issue... I was wondering if relying on dynamic fields could be an option... Something like : * field_name: field_type * customer : string * *_field_a1 : type_a * *_field_a2 : type_a * *_field_b1 : type_b * ... And the prefix each field by the customer name, so for customer1, indexed documents are as follow : * customer : customer1 * customer1_field_a1 : value for field_a1 * customer1_field_a2 : value for field_a2 * customer1_field_b1 : value for field_b1 * ... And for customer2 : * customer : customer2 * customer2_field_a1 : value for field_a1 * customer2_field_a2 : value for field_a2 * customer2_field_b1 : value for field_b1 * ... This solution is simple, and helps isolating each customers fields so features like suggester, spellcheck, ..., things relying on frequencies would work (as if in a single core) I just don't how well does solr scale if the number of fields increases... Then scaling could be achieved depending on number of doc / customer and number of customer / core (if amount of fields consumes resources) Could that help ? -- Tanguy 2012/9/25 Toke Eskildsen t...@statsbiblioteket.dk On Tue, 2012-09-25 at 04:21 +0200, 韦震宇 wrote: The company I'm working in have a website to server more than 10 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer. How many of the customers are active at any given time and how large are the indexes? Depending on usage you might be able to have a limited number of indexes open at any given time and opening new indexes on demand.
Re: How can I create about 100000 independent indexes in Solr?
Hi, Tanguy Oh, I understand now. I don't have the issue as you. Though there are so many customers in our site, but the fields they owned are same. so few field fields are ok in my scene. Best Regards! Monton - Original Message - From: Tanguy Moal tanguy.m...@gmail.com To: solr-user@lucene.apache.org Sent: Thursday, September 27, 2012 4:34 PM Subject: Re: How can I create about 10 independent indexes in Solr? Hello Monton, I wanted to make sure that you understood me well : I really don't how well does solr scale if the number of fields increases... What I mean here is that the more distinct fields you index, the more memory you will need. So if in your schema, you have something like 15 fields declared, then storing data for 100 distinct customers would generate 1500 fields in the index. I really don't know how well would that scale. The simplest solution is one core per customer but the same issue (memory consumption) will rise at some point, I guess. There must be a clever way to do that... -- Tanguy 2012/9/26 韦震宇 weizhe...@win-trust.com Hi, Tanguy I would do as your suggestion. Best Regards! Monton - Original Message - From: Tanguy Moal tanguy.m...@gmail.com To: solr-user@lucene.apache.org; t...@statsbiblioteket.dk Sent: Tuesday, September 25, 2012 11:05 PM Subject: Re: How can I create about 10 independent indexes in Solr? That is an interesting issue... I was wondering if relying on dynamic fields could be an option... Something like : * field_name: field_type * customer : string * *_field_a1 : type_a * *_field_a2 : type_a * *_field_b1 : type_b * ... And the prefix each field by the customer name, so for customer1, indexed documents are as follow : * customer : customer1 * customer1_field_a1 : value for field_a1 * customer1_field_a2 : value for field_a2 * customer1_field_b1 : value for field_b1 * ... And for customer2 : * customer : customer2 * customer2_field_a1 : value for field_a1 * customer2_field_a2 : value for field_a2 * customer2_field_b1 : value for field_b1 * ... This solution is simple, and helps isolating each customers fields so features like suggester, spellcheck, ..., things relying on frequencies would work (as if in a single core) I just don't how well does solr scale if the number of fields increases... Then scaling could be achieved depending on number of doc / customer and number of customer / core (if amount of fields consumes resources) Could that help ? -- Tanguy 2012/9/25 Toke Eskildsen t...@statsbiblioteket.dk On Tue, 2012-09-25 at 04:21 +0200, 韦震宇 wrote: The company I'm working in have a website to server more than 10 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer. How many of the customers are active at any given time and how large are the indexes? Depending on usage you might be able to have a limited number of indexes open at any given time and opening new indexes on demand.
Re: How can I create about 100000 independent indexes in Solr?
Hi,Otis Thank you for your suggestion! I'm considering the a single index solution. Best Regards! Monton - Original Message - From: Otis Gospodnetic otis.gospodne...@gmail.com To: solr-user@lucene.apache.org Sent: Wednesday, September 26, 2012 10:26 AM Subject: Re: How can I create about 10 independent indexes in Solr? Hi, Have a look at ElasticSearch. While it's probably no better at handling 100K cores than Solr it, ES has document and query routing functionality that can help you route documents and queries to specific shards, thus avoiding needing to query all 100K users' data. I think Solr doesn't have this feature yet, but I'm sure it will in the future. Otis -- Search Analytics - http://sematext.com/search-analytics/index.html Performance Monitoring - http://sematext.com/spm/index.html On Tue, Sep 25, 2012 at 10:20 PM, 韦震宇 weizhe...@win-trust.com wrote: Hi,Tom I'm prefer performance, so if the performance of single index solution is better than mutilcore solution, I would choice the former. Best Regards! - Original Message - From: Tom Mortimer tom.m.f...@gmail.com To: solr-user@lucene.apache.org Sent: Tuesday, September 25, 2012 6:06 PM Subject: Re: How can I create about 10 independent indexes in Solr? Hi, Why do you think that the indexes should be independent? What would be the problem with using a single index and filter queries? Tom On 25 Sep 2012, at 03:21, 韦震宇 weizhe...@win-trust.com wrote: Dear all, The company I'm working in have a website to server more than 10 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer. The site http://wiki.apache.org/solr/MultipleIndexes give some solution to create multiple indexes. I want to use multicore solution. But i'm afraid that Solr can't support so many indexes in this solution. The other solution Flattening data into a single index is a choice, but i think it's best to keep all indexes indepent. Could you tell me how to create about 10 independent indexes in Solr? Thank you all for reply!
Re: How can I create about 100000 independent indexes in Solr?
Hi, Why do you think that the indexes should be independent? What would be the problem with using a single index and filter queries? Tom On 25 Sep 2012, at 03:21, 韦震宇 weizhe...@win-trust.com wrote: Dear all, The company I'm working in have a website to server more than 10 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer. The site http://wiki.apache.org/solr/MultipleIndexes give some solution to create multiple indexes. I want to use multicore solution. But i'm afraid that Solr can't support so many indexes in this solution. The other solution Flattening data into a single index is a choice, but i think it's best to keep all indexes indepent. Could you tell me how to create about 10 independent indexes in Solr? Thank you all for reply!
Re: How can I create about 100000 independent indexes in Solr?
On Tue, 2012-09-25 at 12:06 +0200, Tom Mortimer wrote: Why do you think that the indexes should be independent? What would be the problem with using a single index and filter queries? Ranking will be done with regard to the full corpus instead of just the single customer. Whether that is an issue or not depends on the content. e.g. for email archives, the single index will not work very well. - Toke Eskildsen, State and University Library, Denmark
AW: How can I create about 100000 independent indexes in Solr?
Hi weizhenyu! (can you write down your name in roman latters) We also tried to provide dedicated cores for each customer in one of our early system design to get maximum config possibilities. 40k Webshops = 40k Solr Cores, which didn't work out. Problems we encountered: - Threads: Solr create one thread to interact with each core in 3.x of solr , 40k therads will kill any server performance - FileSystem: Each Core needed number of files 30 -100 files depending on settings etc. 40.000 x 100 file handles was not an option for the CentOS (and most other OS) - Managing 100k configs is problem of its own You may look for other solution e.g. split the user base in small number of Cores by use case and try to cover their needs. (btw the solrconfig.xml, schema.xml ultra flexible most likely the cover 95% of your requirements) Regards, Christian Bordis -Ursprüngliche Nachricht- Von: 韦震宇 [mailto:weizhe...@win-trust.com] Gesendet: Dienstag, 25. September 2012 04:21 An: solr-user@lucene.apache.org Betreff: How can I create about 10 independent indexes in Solr? Dear all, The company I'm working in have a website to server more than 10 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer. The site http://wiki.apache.org/solr/MultipleIndexes give some solution to create multiple indexes. I want to use multicore solution. But i'm afraid that Solr can't support so many indexes in this solution. The other solution Flattening data into a single index is a choice, but i think it's best to keep all indexes indepent. Could you tell me how to create about 10 independent indexes in Solr? Thank you all for reply!
Re: How can I create about 100000 independent indexes in Solr?
On Tue, 2012-09-25 at 04:21 +0200, 韦震宇 wrote: The company I'm working in have a website to server more than 10 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer. How many of the customers are active at any given time and how large are the indexes? Depending on usage you might be able to have a limited number of indexes open at any given time and opening new indexes on demand.
Re: How can I create about 100000 independent indexes in Solr?
That is an interesting issue... I was wondering if relying on dynamic fields could be an option... Something like : * field_name: field_type * customer : string * *_field_a1 : type_a * *_field_a2 : type_a * *_field_b1 : type_b * ... And the prefix each field by the customer name, so for customer1, indexed documents are as follow : * customer : customer1 * customer1_field_a1 : value for field_a1 * customer1_field_a2 : value for field_a2 * customer1_field_b1 : value for field_b1 * ... And for customer2 : * customer : customer2 * customer2_field_a1 : value for field_a1 * customer2_field_a2 : value for field_a2 * customer2_field_b1 : value for field_b1 * ... This solution is simple, and helps isolating each customers fields so features like suggester, spellcheck, ..., things relying on frequencies would work (as if in a single core) I just don't how well does solr scale if the number of fields increases... Then scaling could be achieved depending on number of doc / customer and number of customer / core (if amount of fields consumes resources) Could that help ? -- Tanguy 2012/9/25 Toke Eskildsen t...@statsbiblioteket.dk On Tue, 2012-09-25 at 04:21 +0200, 韦震宇 wrote: The company I'm working in have a website to server more than 10 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer. How many of the customers are active at any given time and how large are the indexes? Depending on usage you might be able to have a limited number of indexes open at any given time and opening new indexes on demand.
Re: How can I create about 100000 independent indexes in Solr?
Hi,Tom I'm prefer performance, so if the performance of single index solution is better than mutilcore solution, I would choice the former. Best Regards! - Original Message - From: Tom Mortimer tom.m.f...@gmail.com To: solr-user@lucene.apache.org Sent: Tuesday, September 25, 2012 6:06 PM Subject: Re: How can I create about 10 independent indexes in Solr? Hi, Why do you think that the indexes should be independent? What would be the problem with using a single index and filter queries? Tom On 25 Sep 2012, at 03:21, 韦震宇 weizhe...@win-trust.com wrote: Dear all, The company I'm working in have a website to server more than 10 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer. The site http://wiki.apache.org/solr/MultipleIndexes give some solution to create multiple indexes. I want to use multicore solution. But i'm afraid that Solr can't support so many indexes in this solution. The other solution Flattening data into a single index is a choice, but i think it's best to keep all indexes indepent. Could you tell me how to create about 10 independent indexes in Solr? Thank you all for reply!
Re: How can I create about 100000 independent indexes in Solr?
Hi, Have a look at ElasticSearch. While it's probably no better at handling 100K cores than Solr it, ES has document and query routing functionality that can help you route documents and queries to specific shards, thus avoiding needing to query all 100K users' data. I think Solr doesn't have this feature yet, but I'm sure it will in the future. Otis -- Search Analytics - http://sematext.com/search-analytics/index.html Performance Monitoring - http://sematext.com/spm/index.html On Tue, Sep 25, 2012 at 10:20 PM, 韦震宇 weizhe...@win-trust.com wrote: Hi,Tom I'm prefer performance, so if the performance of single index solution is better than mutilcore solution, I would choice the former. Best Regards! - Original Message - From: Tom Mortimer tom.m.f...@gmail.com To: solr-user@lucene.apache.org Sent: Tuesday, September 25, 2012 6:06 PM Subject: Re: How can I create about 10 independent indexes in Solr? Hi, Why do you think that the indexes should be independent? What would be the problem with using a single index and filter queries? Tom On 25 Sep 2012, at 03:21, 韦震宇 weizhe...@win-trust.com wrote: Dear all, The company I'm working in have a website to server more than 10 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer. The site http://wiki.apache.org/solr/MultipleIndexes give some solution to create multiple indexes. I want to use multicore solution. But i'm afraid that Solr can't support so many indexes in this solution. The other solution Flattening data into a single index is a choice, but i think it's best to keep all indexes indepent. Could you tell me how to create about 10 independent indexes in Solr? Thank you all for reply!
Re: How can I create about 100000 independent indexes in Solr?
Hi, Bordis Thank you very much for your precious experience! I would choice the single index solution. I'm sorry my name is not pronounced good in english ,you would call me monton. Best Regards! Monton - Original Message - From: Christian Bordis c.bor...@epages.com To: solr-user@lucene.apache.org Sent: Tuesday, September 25, 2012 9:18 PM Subject: AW: How can I create about 10 independent indexes in Solr? Hi weizhenyu! (can you write down your name in roman latters) We also tried to provide dedicated cores for each customer in one of our early system design to get maximum config possibilities. 40k Webshops = 40k Solr Cores, which didn't work out. Problems we encountered: - Threads: Solr create one thread to interact with each core in 3.x of solr , 40k therads will kill any server performance - FileSystem: Each Core needed number of files 30 -100 files depending on settings etc. 40.000 x 100 file handles was not an option for the CentOS (and most other OS) - Managing 100k configs is problem of its own You may look for other solution e.g. split the user base in small number of Cores by use case and try to cover their needs. (btw the solrconfig.xml, schema.xml ultra flexible most likely the cover 95% of your requirements) Regards, Christian Bordis -Ursprüngliche Nachricht- Von: 韦震宇 [mailto:weizhe...@win-trust.com] Gesendet: Dienstag, 25. September 2012 04:21 An: solr-user@lucene.apache.org Betreff: How can I create about 10 independent indexes in Solr? Dear all, The company I'm working in have a website to server more than 10 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer. The site http://wiki.apache.org/solr/MultipleIndexes give some solution to create multiple indexes. I want to use multicore solution. But i'm afraid that Solr can't support so many indexes in this solution. The other solution Flattening data into a single index is a choice, but i think it's best to keep all indexes indepent. Could you tell me how to create about 10 independent indexes in Solr? Thank you all for reply!
Re: How can I create about 100000 independent indexes in Solr?
Hi, Eskildsen The customer is company, so the active user is tenfold more than the customers. I'm choicing the a single index and splitting index into multiple shards. Best Regards! Monton - Original Message - From: Toke Eskildsen t...@statsbiblioteket.dk To: solr-user@lucene.apache.org Sent: Tuesday, September 25, 2012 10:37 PM Subject: Re: How can I create about 10 independent indexes in Solr? On Tue, 2012-09-25 at 04:21 +0200, 韦震宇 wrote: The company I'm working in have a website to server more than 10 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer. How many of the customers are active at any given time and how large are the indexes? Depending on usage you might be able to have a limited number of indexes open at any given time and opening new indexes on demand.
Re: How can I create about 100000 independent indexes in Solr?
Hi, Tanguy I would do as your suggestion. Best Regards! Monton - Original Message - From: Tanguy Moal tanguy.m...@gmail.com To: solr-user@lucene.apache.org; t...@statsbiblioteket.dk Sent: Tuesday, September 25, 2012 11:05 PM Subject: Re: How can I create about 10 independent indexes in Solr? That is an interesting issue... I was wondering if relying on dynamic fields could be an option... Something like : * field_name: field_type * customer : string * *_field_a1 : type_a * *_field_a2 : type_a * *_field_b1 : type_b * ... And the prefix each field by the customer name, so for customer1, indexed documents are as follow : * customer : customer1 * customer1_field_a1 : value for field_a1 * customer1_field_a2 : value for field_a2 * customer1_field_b1 : value for field_b1 * ... And for customer2 : * customer : customer2 * customer2_field_a1 : value for field_a1 * customer2_field_a2 : value for field_a2 * customer2_field_b1 : value for field_b1 * ... This solution is simple, and helps isolating each customers fields so features like suggester, spellcheck, ..., things relying on frequencies would work (as if in a single core) I just don't how well does solr scale if the number of fields increases... Then scaling could be achieved depending on number of doc / customer and number of customer / core (if amount of fields consumes resources) Could that help ? -- Tanguy 2012/9/25 Toke Eskildsen t...@statsbiblioteket.dk On Tue, 2012-09-25 at 04:21 +0200, 韦震宇 wrote: The company I'm working in have a website to server more than 10 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer. How many of the customers are active at any given time and how large are the indexes? Depending on usage you might be able to have a limited number of indexes open at any given time and opening new indexes on demand.
How can I create about 100000 independent indexes in Solr?
Dear all, The company I'm working in have a website to server more than 10 customers, and every customer should have it's own search cataegory. So I should create independent index for every customer. The site http://wiki.apache.org/solr/MultipleIndexes give some solution to create multiple indexes. I want to use multicore solution. But i'm afraid that Solr can't support so many indexes in this solution. The other solution Flattening data into a single index is a choice, but i think it's best to keep all indexes indepent. Could you tell me how to create about 10 independent indexes in Solr? Thank you all for reply!
