Add Entry to Support Page
Solr Community, I'm Christoph Schmidt (http://www.moresophy.com/de/management), CEO of the german company moresophy GmbH. My Solr Wiki name is: - ChristophSchmidt We are working with Lucene since 2003 and Solr 2012 and are building linguistic token filters and plugins for Solr. We would like to add the following entry to the Solr Support page: moresophy GmbHhttp://www.moresophy.com/: consulting in Lucene, Solr, elasticsearch, specialization in linguistic and semantic enrichment and high scalability content clouds (DE/AT/CH) a class=mailto href=mailto:i...@moresophy.com;i...@moresophy.com/amailto:i...@moresophy.com%3c/a Best regards Christoph Schmidt ___ Dr. Christoph Schmidt | Geschäftsführer P +49-89-523041-72 M +49-171-1419367 Skype: cs_moresophy christoph.schm...@moresophy.demailto:heiko.be...@moresophy.de www.moresophy.comhttp://www.moresophy.com/ moresophy GmbH | Fraunhoferstrasse 15 | 82152 München-Martinsried Handelsregister beim Amtsgericht München, NR. HRB 136075 Umsatzsteueridentifikationsnummer: DE813188826 Vertreten durch die Geschäftsführer: Prof. Dr. Heiko Beier | Dr. Christoph Schmidt Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden. [cid:890335816@02112010-31c2]
AW: Scaling to large Number of Collections
Is there a Jira task for this? Thanks Christoph -Ursprüngliche Nachricht- Von: Mark Miller [mailto:markrmil...@gmail.com] Gesendet: Sonntag, 31. August 2014 14:24 An: solr-user Betreff: Re: Scaling to large Number of Collections On Aug 31, 2014, at 4:04 AM, Christoph Schmidt christoph.schm...@moresophy.de wrote: we see at least two problems when scaling to large number of collections. I would like to ask the community, if they are known and maybe already addressed in development: We have a SolrCloud running with the following numbers: - 5 Servers (each 24 CPUs, 128 RAM) - 13.000 Collection with 25.000 SolrCores in the Cloud The Cloud is working fine, but we see two problems, if we like to scale further 1. Resource consumption of native system threads We see that each collection opens at least two threads: one for the zookeeper (coreZkRegister-1-thread-5154) and one for the searcher (searcherExecutor-28357-thread-1) We will run in OutOfMemoryError: unable to create new native thread. Maybe the architecture could be changed here to use thread pools? 2. The shutdown and the startup of one server in the SolrCloud takes 2 hours. So a rolling start is about 10h. For me the problem seems to be that leader election is linear. The Overseer does core per core. The organisation of the cloud is not done parallel or distributed. Is this already addressed by https://issues.apache.org/jira/browse/SOLR-5473 or is there more needed? 2. No, but it should have been fixed by another issue that will be in 4.10. - Mark http://about.me/markrmiller
AW: Scaling to large Number of Collections
Yes, we will think about this how to reorganise the application. Thanks Christoph -Ursprüngliche Nachricht- Von: Joseph Obernberger [mailto:joseph.obernber...@gmail.com] Gesendet: Sonntag, 31. August 2014 16:58 An: solr-user@lucene.apache.org Betreff: Re: Scaling to large Number of Collections Could you add another field(s) to your application and use that instead of creating collections/cores? When you execute a search, instead of picking a core, just search a single large core but add in a field which contains some core ID. -Joe http://www.lovehorsepower.com On Sun, Aug 31, 2014 at 8:23 AM, Mark Miller markrmil...@gmail.com wrote: On Aug 31, 2014, at 4:04 AM, Christoph Schmidt christoph.schm...@moresophy.de wrote: we see at least two problems when scaling to large number of collections. I would like to ask the community, if they are known and maybe already addressed in development: We have a SolrCloud running with the following numbers: - 5 Servers (each 24 CPUs, 128 RAM) - 13.000 Collection with 25.000 SolrCores in the Cloud The Cloud is working fine, but we see two problems, if we like to scale further 1. Resource consumption of native system threads We see that each collection opens at least two threads: one for the zookeeper (coreZkRegister-1-thread-5154) and one for the searcher (searcherExecutor-28357-thread-1) We will run in OutOfMemoryError: unable to create new native thread. Maybe the architecture could be changed here to use thread pools? 2. The shutdown and the startup of one server in the SolrCloud takes 2 hours. So a rolling start is about 10h. For me the problem seems to be that leader election is linear. The Overseer does core per core. The organisation of the cloud is not done parallel or distributed. Is this already addressed by https://issues.apache.org/jira/browse/SOLR-5473 or is there more needed? 2. No, but it should have been fixed by another issue that will be in 4.10. - Mark http://about.me/markrmiller
AW: Scaling to large Number of Collections
Hm, temporarily more threads is hard. We already reduced -Xss256k. Wouldn't it be better to use Callable and Executor as proposed in: http://stackoverflow.com/questions/16789288/java-lang-outofmemoryerror-unable-to-create-new-native-thread and limit the number of used threads to the number of CPUs or twice or triple of it? Native threads are restricted by the total virtual memory of the system (at least in linux as far as I know). So the 10.000 threads, we use, is somehow near the limit of the hardware, we have. Christoph -Ursprüngliche Nachricht- Von: Ramkumar R. Aiyengar [mailto:andyetitmo...@gmail.com] Gesendet: Sonntag, 31. August 2014 17:53 An: solr-user@lucene.apache.org Betreff: Re: Scaling to large Number of Collections On 31 Aug 2014 13:24, Mark Miller markrmil...@gmail.com wrote: On Aug 31, 2014, at 4:04 AM, Christoph Schmidt christoph.schm...@moresophy.de wrote: we see at least two problems when scaling to large number of collections. I would like to ask the community, if they are known and maybe already addressed in development: We have a SolrCloud running with the following numbers: - 5 Servers (each 24 CPUs, 128 RAM) - 13.000 Collection with 25.000 SolrCores in the Cloud The Cloud is working fine, but we see two problems, if we like to scale further 1. Resource consumption of native system threads We see that each collection opens at least two threads: one for the zookeeper (coreZkRegister-1-thread-5154) and one for the searcher (searcherExecutor-28357-thread-1) We will run in OutOfMemoryError: unable to create new native thread. Maybe the architecture could be changed here to use thread pools? 2. The shutdown and the startup of one server in the SolrCloud takes 2 hours. So a rolling start is about 10h. For me the problem seems to be that leader election is linear. The Overseer does core per core. The organisation of the cloud is not done parallel or distributed. Is this already addressed by https://issues.apache.org/jira/browse/SOLR-5473 or is there more needed? 2. No, but it should have been fixed by another issue that will be in 4.10. Note however that this fix will result in even more temporary thread usage as all leadership elections will happen in parallel, so you might still end up with these out of threads issue again. Quite possibly the out of threads issue is just some system soft limit which is kicking in. Linux certainly has a limit you can configure through sysctl, your OS, whatever that might be, probably does the same. May be worth exploring if you can bump that up. - Mark http://about.me/markrmiller
AW: Scaling to large Number of Collections
At peak time a lot of customers will be on the system. Sure not all in the same second but within an hour we see a lot of them. So having something like stand-by cores will help to reduce resource consumption (memory, threads) if the core are rarely used. Doing a warmup at login time of the user and a kind of release at logoff. But I don't think this will help us now. The denormalization is also hard, because as you said our data will explode. No, I think the reduction of native threads will help or a restructuring of the application architecture by adding a new user-id field and combining user collections. Best Christoph -Ursprüngliche Nachricht- Von: Erick Erickson [mailto:erickerick...@gmail.com] Gesendet: Sonntag, 31. August 2014 18:04 An: solr-user@lucene.apache.org Betreff: Re: Scaling to large Number of Collections What is your access pattern? By that I mean do all the cores need to be searched at the same time or is it reasonable for them to be loaded on demand? This latter would impose the penalty of the first time a collection was accessed there would be a delay while the core loaded. I suppose I'm asking how many customers are using the system simultaneously?. One way around that is to fire a dummy query behind the scenes when a user logs on but before she actually executes a search. Why I'm asking: See this page: http://wiki.apache.org/solr/LotsOfCores. It was intended for the multi-tenancy case in which you could count on a subset of users being logged on at once. WARNING! LotsOfCores is NOT supported in SolrCloud at this point! There has been some talk of extending support for SolrCloud, but no action as it's one of those cases that has lots of implications particularly around ZooKeeper knowing the state of all the cores, cores going into recovery in a cascading fashionetc. It's not at all clear that it _can_ be extended to SolrCloud for that matter without doing great violence to the code. With the LotsOfCores approach (and assuming somebody volunteers to code it up), the number of cores hosted on a particular node can be many thousands. The limits will come from how many of them have to be up and running simultaneously. The limits would come from two places: 1 The time it takes to recursively walk your SOLR_HOME directory and discover the cores (I see about 1,000 cores/second discovered on my laptop, admittedly an SSD, and there has been no optimization done to this process). 2 having to keep a table of all the cores and their information (home directory and the like) in memory, but practically I don't think this is a problem. I haven't actually measured, but the size of each entry is almost certainly less than 1K and probably closer to 0.5K. But it really does bring us back to the question of whether all these cores are necessary or not. The usual technique for handling this with the LotsOfCores option is to combine the records into a number of smaller cores. Without knowing your requirements in detail, something like a customers core and a products core where, say, each product has a field with tokens indicating what users had access or vice versa, and (possibly) using pseudo joins. In one view, this is an ACL problem which has several solutions, each with drawbacks of course. Or just de-normalizing your data entirely and just have a core per customer with _all_ the products indexed in to it. Like I said, I don't know enough details to have a clue whether the data would explode unacceptably. Anyway, enough on a Sunday morning! Best, Erick On Sun, Aug 31, 2014 at 8:18 AM, Shawn Heisey s...@elyograg.org wrote: On 8/31/2014 8:58 AM, Joseph Obernberger wrote: Could you add another field(s) to your application and use that instead of creating collections/cores? When you execute a search, instead of picking a core, just search a single large core but add in a field which contains some core ID. This is a nice idea. Have one big collection in your cloud and use an additional field in your queries to filter down to a specific user's data. It'd be really nice to write a custom search component that ensures there is a filter query for that specific field, and if it's not present, change the search results to include a document that informs the caller that they're not doing it right. http://www.portal2sounds.com/1780 (That URL probably won't work correctly on mobile browsers) Thanks, Shawn
AW: Scaling to large Number of Collections
Yes, this would help us in our scenario. -Ursprüngliche Nachricht- Von: Jack Krupansky [mailto:j...@basetechnology.com] Gesendet: Sonntag, 31. August 2014 18:10 An: solr-user@lucene.apache.org Betreff: Re: Scaling to large Number of Collections We should also consider lightly-sharded collections. IOW, even if a cluster has dozens or a hundred nodes or more, the goal may not be to shard all collections across all shards, which is fine for the really large collections, but to also support collections which may only need to be sharded for a few shards or even just a single shard, and to instead focus the attention on large number of collections rather than heavily-sharded collections. -- Jack Krupansky -Original Message- From: Erick Erickson Sent: Sunday, August 31, 2014 12:04 PM To: solr-user@lucene.apache.org Subject: Re: Scaling to large Number of Collections What is your access pattern? By that I mean do all the cores need to be searched at the same time or is it reasonable for them to be loaded on demand? This latter would impose the penalty of the first time a collection was accessed there would be a delay while the core loaded. I suppose I'm asking how many customers are using the system simultaneously?. One way around that is to fire a dummy query behind the scenes when a user logs on but before she actually executes a search. Why I'm asking: See this page: http://wiki.apache.org/solr/LotsOfCores. It was intended for the multi-tenancy case in which you could count on a subset of users being logged on at once. WARNING! LotsOfCores is NOT supported in SolrCloud at this point! There has been some talk of extending support for SolrCloud, but no action as it's one of those cases that has lots of implications particularly around ZooKeeper knowing the state of all the cores, cores going into recovery in a cascading fashionetc. It's not at all clear that it _can_ be extended to SolrCloud for that matter without doing great violence to the code. With the LotsOfCores approach (and assuming somebody volunteers to code it up), the number of cores hosted on a particular node can be many thousands. The limits will come from how many of them have to be up and running simultaneously. The limits would come from two places: 1 The time it takes to recursively walk your SOLR_HOME directory and discover the cores (I see about 1,000 cores/second discovered on my laptop, admittedly an SSD, and there has been no optimization done to this process). 2 having to keep a table of all the cores and their information (home directory and the like) in memory, but practically I don't think this is a problem. I haven't actually measured, but the size of each entry is almost certainly less than 1K and probably closer to 0.5K. But it really does bring us back to the question of whether all these cores are necessary or not. The usual technique for handling this with the LotsOfCores option is to combine the records into a number of smaller cores. Without knowing your requirements in detail, something like a customers core and a products core where, say, each product has a field with tokens indicating what users had access or vice versa, and (possibly) using pseudo joins. In one view, this is an ACL problem which has several solutions, each with drawbacks of course. Or just de-normalizing your data entirely and just have a core per customer with _all_ the products indexed in to it. Like I said, I don't know enough details to have a clue whether the data would explode unacceptably. Anyway, enough on a Sunday morning! Best, Erick On Sun, Aug 31, 2014 at 8:18 AM, Shawn Heisey s...@elyograg.org wrote: On 8/31/2014 8:58 AM, Joseph Obernberger wrote: Could you add another field(s) to your application and use that instead of creating collections/cores? When you execute a search, instead of picking a core, just search a single large core but add in a field which contains some core ID. This is a nice idea. Have one big collection in your cloud and use an additional field in your queries to filter down to a specific user's data. It'd be really nice to write a custom search component that ensures there is a filter query for that specific field, and if it's not present, change the search results to include a document that informs the caller that they're not doing it right. http://www.portal2sounds.com/1780 (That URL probably won't work correctly on mobile browsers) Thanks, Shawn
AW: Scaling to large Number of Collections
We already reduced the -Xss256k. How could we reduce the size of the transaction log? By less autoCommits? Or could it be cleaned up? Thanks Christoph -Ursprüngliche Nachricht- Von: Shalin Shekhar Mangar [mailto:shalinman...@gmail.com] Gesendet: Sonntag, 31. August 2014 20:12 An: solr-user@lucene.apache.org; Mark Miller Betreff: Re: Scaling to large Number of Collections Yeah, I second Mark's suggestion on reducing the stack size. The default on modern 64-bit boxes is usually 1024KB which adds up to a lot when you're running 5000 cores (5000 * 2 = 1MB). I think the zk register thread can be pooled together but the search threads can't be because we'd run into deadlock situations. I'll have to think more on that. As for your 2nd question on slow restarts - make sure you tune autoCommit settings so that your transaction logs don't get so big. When people complain about slow restarts, large transaction logs are usually the culprit. As for the larger questions about lot of collections, yeah, I think you'll see more work happening in that direction. We, at Lucidworks, have been spending quite a bit of time making it work well with SolrCloud. On Sun, Aug 31, 2014 at 9:39 AM, Mark Miller markrmil...@gmail.com wrote: so you might still end up with these out of threads issue again. You can also generally drop the stack size (Xss) quite a bit to to handle more threads. Beyond that, there are some thread pools you can configure. However, until we fix the distrib deadlock issue, you don't want to drop the container thread pool too much. There are other control points though. - Mark http://about.me/markrmiller On Sun, Aug 31, 2014 at 11:53 AM, Ramkumar R. Aiyengar andyetitmo...@gmail.com wrote: On 31 Aug 2014 13:24, Mark Miller markrmil...@gmail.com wrote: On Aug 31, 2014, at 4:04 AM, Christoph Schmidt christoph.schm...@moresophy.de wrote: we see at least two problems when scaling to large number of collections. I would like to ask the community, if they are known and maybe already addressed in development: We have a SolrCloud running with the following numbers: - 5 Servers (each 24 CPUs, 128 RAM) - 13.000 Collection with 25.000 SolrCores in the Cloud The Cloud is working fine, but we see two problems, if we like to scale further 1. Resource consumption of native system threads We see that each collection opens at least two threads: one for the zookeeper (coreZkRegister-1-thread-5154) and one for the searcher (searcherExecutor-28357-thread-1) We will run in OutOfMemoryError: unable to create new native thread. Maybe the architecture could be changed here to use thread pools? 2. The shutdown and the startup of one server in the SolrCloud takes 2 hours. So a rolling start is about 10h. For me the problem seems to be that leader election is linear. The Overseer does core per core. The organisation of the cloud is not done parallel or distributed. Is this already addressed by https://issues.apache.org/jira/browse/SOLR-5473 or is there more needed? 2. No, but it should have been fixed by another issue that will be in 4.10. Note however that this fix will result in even more temporary thread usage as all leadership elections will happen in parallel, so you might still end up with these out of threads issue again. Quite possibly the out of threads issue is just some system soft limit which is kicking in. Linux certainly has a limit you can configure through sysctl, your OS, whatever that might be, probably does the same. May be worth exploring if you can bump that up. - Mark http://about.me/markrmiller -- Regards, Shalin Shekhar Mangar.
AW: AW: AW: Scaling to large Number of Collections
Isn't this not just a question of configuration and my hardware. The better my hardware the more cores I can keep in memory. Let's say I can keep 10.000 cores in memory, when I reach the limit, I will put the last recent used to dormant and wakeup the new one I need. So I can scale further by reuse of memory, threads, etc. paying the price of wakeup the cores. Isn't it just reopen the reader and searcher? How big is the solr overhead? -Ursprüngliche Nachricht- Von: Jack Krupansky [mailto:j...@basetechnology.com] Gesendet: Montag, 1. September 2014 14:50 An: solr-user@lucene.apache.org Betreff: Re: AW: Scaling to large Number of Collections And I would add another suggested requirement - dormant collections - collections which may once have been active, but have not seen any recent activity and can hence be suspended or swapped out until such time as activity resumes and they can then be reactivated or reloaded. That inactivity threshold might be something like an hour, but should be configurable globally and per-collection. The alternative is an application server which maintains that activity state and starts up and shuts down discrete Solr server instances for each tenant's collection(s). This raises the question: How many of your collections need to be simultaneously active? Say, in a one-hour period, how many of them will be updating and serving queries, and what query load per-collection and total query load do you need to design for? -- Jack Krupansky -Original Message- From: Christoph Schmidt Sent: Monday, September 1, 2014 3:50 AM To: solr-user@lucene.apache.org Subject: AW: Scaling to large Number of Collections Yes, this would help us in our scenario. -Ursprüngliche Nachricht- Von: Jack Krupansky [mailto:j...@basetechnology.com] Gesendet: Sonntag, 31. August 2014 18:10 An: solr-user@lucene.apache.org Betreff: Re: Scaling to large Number of Collections We should also consider lightly-sharded collections. IOW, even if a cluster has dozens or a hundred nodes or more, the goal may not be to shard all collections across all shards, which is fine for the really large collections, but to also support collections which may only need to be sharded for a few shards or even just a single shard, and to instead focus the attention on large number of collections rather than heavily-sharded collections. -- Jack Krupansky -Original Message- From: Erick Erickson Sent: Sunday, August 31, 2014 12:04 PM To: solr-user@lucene.apache.org Subject: Re: Scaling to large Number of Collections What is your access pattern? By that I mean do all the cores need to be searched at the same time or is it reasonable for them to be loaded on demand? This latter would impose the penalty of the first time a collection was accessed there would be a delay while the core loaded. I suppose I'm asking how many customers are using the system simultaneously?. One way around that is to fire a dummy query behind the scenes when a user logs on but before she actually executes a search. Why I'm asking: See this page: http://wiki.apache.org/solr/LotsOfCores. It was intended for the multi-tenancy case in which you could count on a subset of users being logged on at once. WARNING! LotsOfCores is NOT supported in SolrCloud at this point! There has been some talk of extending support for SolrCloud, but no action as it's one of those cases that has lots of implications particularly around ZooKeeper knowing the state of all the cores, cores going into recovery in a cascading fashionetc. It's not at all clear that it _can_ be extended to SolrCloud for that matter without doing great violence to the code. With the LotsOfCores approach (and assuming somebody volunteers to code it up), the number of cores hosted on a particular node can be many thousands. The limits will come from how many of them have to be up and running simultaneously. The limits would come from two places: 1 The time it takes to recursively walk your SOLR_HOME directory and discover the cores (I see about 1,000 cores/second discovered on my laptop, admittedly an SSD, and there has been no optimization done to this process). 2 having to keep a table of all the cores and their information (home directory and the like) in memory, but practically I don't think this is a problem. I haven't actually measured, but the size of each entry is almost certainly less than 1K and probably closer to 0.5K. But it really does bring us back to the question of whether all these cores are necessary or not. The usual technique for handling this with the LotsOfCores option is to combine the records into a number of smaller cores. Without knowing your requirements in detail, something like a customers core and a products core where, say, each product has a field with tokens indicating what users had access or vice versa, and (possibly) using pseudo joins. In one view, this is an ACL problem which has several
Scaling to large Number of Collections
we see at least two problems when scaling to large number of collections. I would like to ask the community, if they are known and maybe already addressed in development: We have a SolrCloud running with the following numbers: - 5 Servers (each 24 CPUs, 128 RAM) - 13.000 Collection with 25.000 SolrCores in the Cloud The Cloud is working fine, but we see two problems, if we like to scale further 1. Resource consumption of native system threads We see that each collection opens at least two threads: one for the zookeeper (coreZkRegister-1-thread-5154) and one for the searcher (searcherExecutor-28357-thread-1) We will run in OutOfMemoryError: unable to create new native thread. Maybe the architecture could be changed here to use thread pools? 2. The shutdown and the startup of one server in the SolrCloud takes 2 hours. So a rolling start is about 10h. For me the problem seems to be that leader election is linear. The Overseer does core per core. The organisation of the cloud is not done parallel or distributed. Is this already addressed by https://issues.apache.org/jira/browse/SOLR-5473 or is there more needed? Thanks for discussion and help Christoph ___ Dr. Christoph Schmidt | Geschäftsführer P +49-89-523041-72 M +49-171-1419367 Skype: cs_moresophy christoph.schm...@moresophy.demailto:heiko.be...@moresophy.de www.moresophy.comhttp://www.moresophy.com/ moresophy GmbH | Fraunhoferstrasse 15 | 82152 München-Martinsried
AW: Scaling to large Number of Collections
One collection has 2 replicas, no sharding, the collections are not that big. No, they are unfortunately not independent. There are collections with customer documents (some thousand customers) and product collections. One customer has at least on customer collection and 1 to some hundred products. The combination of these collections is used to drive the search of a Liferay portal. Each customer has its own Liferay portal. We could split up the cluster in several clusters by customers, but then we had for duplicate the product collections in each SolrCluster. Will Solr go in the direction of large number of collections? And the question is, what is a large number? Best Christoph -Ursprüngliche Nachricht- Von: Jack Krupansky [mailto:j...@basetechnology.com] Gesendet: Sonntag, 31. August 2014 14:09 An: solr-user@lucene.apache.org Betreff: Re: Scaling to large Number of Collections How are the 5 servers arranged in terms of shards and replicas? 5 shards with 1 replica each, 1 shard with 5 replicas, 2 shards with 2 and 3 replicas, or... what? How big is each collection? The key strength of SolrCloud is scaling large collections via shards, NOT scaling large numbers of collections. If you have large numbers of collections, maybe they should be divided into separate clusters, especially if they are independent. Is this a multi-tenancy situation or a single humongous app? In any case, large numbers of collections in a single SolrCloud cluster is not a supported scenario at this time. Certainly suggestions for future enhancement can be made though. -- Jack Krupansky -Original Message- From: Christoph Schmidt Sent: Sunday, August 31, 2014 4:04 AM To: solr-user@lucene.apache.org Subject: Scaling to large Number of Collections we see at least two problems when scaling to large number of collections. I would like to ask the community, if they are known and maybe already addressed in development: We have a SolrCloud running with the following numbers: - 5 Servers (each 24 CPUs, 128 RAM) - 13.000 Collection with 25.000 SolrCores in the Cloud The Cloud is working fine, but we see two problems, if we like to scale further 1. Resource consumption of native system threads We see that each collection opens at least two threads: one for the zookeeper (coreZkRegister-1-thread-5154) and one for the searcher (searcherExecutor-28357-thread-1) We will run in OutOfMemoryError: unable to create new native thread. Maybe the architecture could be changed here to use thread pools? 2. The shutdown and the startup of one server in the SolrCloud takes 2 hours. So a rolling start is about 10h. For me the problem seems to be that leader election is linear. The Overseer does core per core. The organisation of the cloud is not done parallel or distributed. Is this already addressed by https://issues.apache.org/jira/browse/SOLR-5473 or is there more needed? Thanks for discussion and help Christoph ___ Dr. Christoph Schmidt | Geschäftsführer P +49-89-523041-72 M +49-171-1419367 Skype: cs_moresophy christoph.schm...@moresophy.demailto:heiko.be...@moresophy.de www.moresophy.comhttp://www.moresophy.com/ moresophy GmbH | Fraunhoferstrasse 15 | 82152 München-Martinsried
