[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926810#comment-15926810 ] Varun Thacker commented on SOLR-7191: - > Restarting a single node takes over two minutes in good

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926204#comment-15926204 ] Joshua Humphries commented on SOLR-7191: Our cluster has many thousands of collections, most of

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925821#comment-15925821 ] Tim Owen commented on SOLR-7191: Admittedly not thousands of collections, but another anecdote. Each of our

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15925790#comment-15925790 ] Jerome Yang commented on SOLR-7191: --- After I test on solr6.4.2 in cloud mode. 32 nodes, on 3 centos6.7

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15893410#comment-15893410 ] Shawn Heisey commented on SOLR-7191: Now that SOLR-10130 has been discovered, there's a possibility

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833804#comment-15833804 ] Damien Kamerman commented on SOLR-7191: --- Regarding the extra threads, I'm thinking the issue is that

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833172#comment-15833172 ] Shawn Heisey commented on SOLR-7191: bq. Shawn where can I raise maxThreads of jetty?

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832908#comment-15832908 ] Yago Riveiro commented on SOLR-7191: Restarting a node in 6.3 now takes forever ... I bumped

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832705#comment-15832705 ] Mark Miller commented on SOLR-7191: --- I know we have gotten pretty good at labeling and grouping threads,

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832668#comment-15832668 ] Shawn Heisey commented on SOLR-7191: With hard nproc at 61440, soft nproc at 40960, and maxThreads for

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832664#comment-15832664 ] Shawn Heisey commented on SOLR-7191: I figured I would try the test setup again on Solr 6.4, see

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15702278#comment-15702278 ] Erick Erickson commented on SOLR-7191: -- [~noble.paul] Can we close this since SOLR-7280 has been

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15357167#comment-15357167 ] Noble Paul commented on SOLR-7191: -- I have simplified this patch and moved it over to SOLR-7280 . I plan

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15357146#comment-15357146 ] Noble Paul commented on SOLR-7191: -- yeah, normally you are fine. If there is a GC pause in the overseer

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353556#comment-15353556 ] Erick Erickson commented on SOLR-7191: -- yeah, probably as I'm out of my depth here. Since each replica

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353502#comment-15353502 ] Scott Blum commented on SOLR-7191: -- [~erickerickson] we may be talking about 2 different things? I'm

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353437#comment-15353437 ] Erick Erickson commented on SOLR-7191: -- FWIW, I tried both and had no trouble even when overseer was

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353362#comment-15353362 ] Erick Erickson commented on SOLR-7191: -- IIUC, the whole watcher thing is replica based. So N replicas

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353293#comment-15353293 ] Noble Paul commented on SOLR-7191: -- The cluster was very stable when I used dedicated overseer using the

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353278#comment-15353278 ] Erick Erickson commented on SOLR-7191: -- Empirically it was fine, but that was on very few runs. I had

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353262#comment-15353262 ] Scott Blum commented on SOLR-7191: -- (for the record, this was on a 5.5.1 based build) > Improve stability

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353261#comment-15353261 ] Scott Blum commented on SOLR-7191: -- Paginated getChildren().. always wondered why that wasn't a thing. >

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353258#comment-15353258 ] Shawn Heisey commented on SOLR-7191: bq. That seems... insane. I am glad that I am not the only one to

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353247#comment-15353247 ] Scott Blum commented on SOLR-7191: -- This may be unrelated to the current patch work, but seems relevant to

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352431#comment-15352431 ] damien kamerman commented on SOLR-7191: --- Concern based on general principles. e.g. if a shard has all

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352399#comment-15352399 ] Noble Paul commented on SOLR-7191: -- bq.f there is a collection that has more than coreLoadThreadCount or

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352349#comment-15352349 ] Erick Erickson commented on SOLR-7191: -- Hmmm, is this a concern based on general principles or on a

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352228#comment-15352228 ] damien kamerman commented on SOLR-7191: --- Only coreLoadThreadCount cores are registering at a time on

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352216#comment-15352216 ] Erick Erickson commented on SOLR-7191: -- Noble: A few comments: > the stress setup I have is sailing

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340698#comment-15340698 ] damien kamerman commented on SOLR-7191: --- The patch from March 2015 was against Solr 4.10. The patch

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340685#comment-15340685 ] Erick Erickson commented on SOLR-7191: -- [~noble.paul] Not sure what version this patch was against. My

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15340106#comment-15340106 ] Noble Paul commented on SOLR-7191: -- which version is the patch built on? > Improve stability and startup

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325626#comment-15325626 ] damien kamerman commented on SOLR-7191: --- This fits with what I've seen on solr 4/5. The cores

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15325575#comment-15325575 ] Erick Erickson commented on SOLR-7191: -- I had to chase after this for a while, so I'm recording

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970255#comment-14970255 ] Damien Kamerman commented on SOLR-7191: --- After 2min around 100 collections all-green. This is with a

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14970253#comment-14970253 ] Damien Kamerman commented on SOLR-7191: --- 1. hmmm cancel that. Initially I noticed very slow (around

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14968920#comment-14968920 ] Shalin Shekhar Mangar commented on SOLR-7191: - Thanks Damien. # What is the purpose of the

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14966893#comment-14966893 ] Shawn Heisey commented on SOLR-7191: bq. I created 6,000 collections (3 nodes; 2 x replicas) and

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14642896#comment-14642896 ] Shawn Heisey commented on SOLR-7191: Another user on the mailing list brought up

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14642990#comment-14642990 ] Shalin Shekhar Mangar commented on SOLR-7191: - bq. Another user on the mailing

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372155#comment-14372155 ] Shalin Shekhar Mangar commented on SOLR-7191: - That's awesome, Damien. I'll

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14372212#comment-14372212 ] Shalin Shekhar Mangar commented on SOLR-7191: - bq. The biggest problem I

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14368557#comment-14368557 ] Damien Kamerman commented on SOLR-7191: --- Shalin, Another change I made was to cache

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14362511#comment-14362511 ] Shalin Shekhar Mangar commented on SOLR-7191: - This is a very broad issue so

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358685#comment-14358685 ] Shawn Heisey commented on SOLR-7191: [~dk]: The first thing I thought when I saw that

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358149#comment-14358149 ] Damien Kamerman commented on SOLR-7191: --- With 10K cores I was still seeing many cores

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354875#comment-14354875 ] Shawn Heisey commented on SOLR-7191: Now that I look closer at that log snippet, it

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354868#comment-14354868 ] Shawn Heisey commented on SOLR-7191: The branch_5x code is a lot more stable than 5.0,

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354965#comment-14354965 ] Shalin Shekhar Mangar commented on SOLR-7191: - bq. I tested 4,000 cores on

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14354379#comment-14354379 ] Damien Kamerman commented on SOLR-7191: --- I tested 4,000 cores on branch_5x and found

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349622#comment-14349622 ] Shawn Heisey commented on SOLR-7191: On the plus side ... in branch_5x, drawing the

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349849#comment-14349849 ] Shawn Heisey commented on SOLR-7191: Followup on the attached log, because I realized

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14349163#comment-14349163 ] Shalin Shekhar Mangar commented on SOLR-7191: - Another issue that will help us

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348836#comment-14348836 ] Shawn Heisey commented on SOLR-7191: As I said above, I started over with a fresh cloud

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347689#comment-14347689 ] Shawn Heisey commented on SOLR-7191: At this time I do not know what the actual issues

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347813#comment-14347813 ] Shawn Heisey commented on SOLR-7191: I have thought about this in terms of deciding

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347834#comment-14347834 ] Shawn Heisey commented on SOLR-7191: Regarding jute.maxbuffer: The new stateFormat

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14347746#comment-14347746 ] Erick Erickson commented on SOLR-7191: -- Just throwing this out there, but tangentially

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348235#comment-14348235 ] Shalin Shekhar Mangar commented on SOLR-7191: - Shawn, can you try with

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348132#comment-14348132 ] Damien Kamerman commented on SOLR-7191: --- I'm more concerned with stability than

[jira] [Commented] (SOLR-7191) Improve stability and startup performance of SolrCloud with thousands of collections

[ https://issues.apache.org/jira/browse/SOLR-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348240#comment-14348240 ] Shalin Shekhar Mangar commented on SOLR-7191: - We can optimize the node