[jira] [Commented] (SOLR-8500) Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients configurable by a system property
[ https://issues.apache.org/jira/browse/SOLR-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135895#comment-15135895 ] ASF subversion and git services commented on SOLR-8500: --- Commit 129f7153087b279908d6340e7f8a5b024f0f7cad in lucene-solr's branch refs/heads/branch_5x from [~erickerickson] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=129f715 ] SOLR-8500: Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients configurable by a system property (cherry picked from commit 3e7fe7867f64b254680d462092d01f07858aa7c3) Conflicts: solr/CHANGES.txt > Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients > configurable by a system property > - > > Key: SOLR-8500 > URL: https://issues.apache.org/jira/browse/SOLR-8500 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-8500.patch > > > Despite the warning in that code, in extremely high throughput situations > where there are guaranteed to be no updates to existing documents, it can be > useful to have more than one runner. > I envision this as an "expert" kind of thing, used only in situations where > the a-priori knowledge is that there are no updates to existing documents. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8500) Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients configurable by a system property
[ https://issues.apache.org/jira/browse/SOLR-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135868#comment-15135868 ] ASF subversion and git services commented on SOLR-8500: --- Commit 3e7fe7867f64b254680d462092d01f07858aa7c3 in lucene-solr's branch refs/heads/master from [~erickerickson] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3e7fe78 ] SOLR-8500: Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients configurable by a system property > Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients > configurable by a system property > - > > Key: SOLR-8500 > URL: https://issues.apache.org/jira/browse/SOLR-8500 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-8500.patch > > > Despite the warning in that code, in extremely high throughput situations > where there are guaranteed to be no updates to existing documents, it can be > useful to have more than one runner. > I envision this as an "expert" kind of thing, used only in situations where > the a-priori knowledge is that there are no updates to existing documents. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8500) Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients configurable by a system property
[ https://issues.apache.org/jira/browse/SOLR-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135896#comment-15135896 ] Erick Erickson commented on SOLR-8500: -- NOTE: This is an "expert" level operation, see the CHANGES.txt entry, reproduced here: this is an expert option and can result in more often needing to do full index replication for recovery, the sweet spot for using this is very high volume, leader-only indexing. > Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients > configurable by a system property > - > > Key: SOLR-8500 > URL: https://issues.apache.org/jira/browse/SOLR-8500 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-8500.patch > > > Despite the warning in that code, in extremely high throughput situations > where there are guaranteed to be no updates to existing documents, it can be > useful to have more than one runner. > I envision this as an "expert" kind of thing, used only in situations where > the a-priori knowledge is that there are no updates to existing documents. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8500) Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients configurable by a system property
[ https://issues.apache.org/jira/browse/SOLR-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135874#comment-15135874 ] ASF subversion and git services commented on SOLR-8500: --- Commit 112a2311df50142ec19ec0033133fbc10df223c9 in lucene-solr's branch refs/heads/master from [~erickerickson] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=112a231 ] Put CHANGES entry for SOLR-8500 in the wrong section. > Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients > configurable by a system property > - > > Key: SOLR-8500 > URL: https://issues.apache.org/jira/browse/SOLR-8500 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-8500.patch > > > Despite the warning in that code, in extremely high throughput situations > where there are guaranteed to be no updates to existing documents, it can be > useful to have more than one runner. > I envision this as an "expert" kind of thing, used only in situations where > the a-priori knowledge is that there are no updates to existing documents. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8500) Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients configurable by a system property
[ https://issues.apache.org/jira/browse/SOLR-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15128647#comment-15128647 ] Erick Erickson commented on SOLR-8500: -- Oh, _that_ kind of reorder As it happens, in this case the volume is so high that they...er...well, can't recover a replica if it gets out of sync, they have to wait for the indexing for that time slice to stop and do a full sync. None of which matters, I see Yonik is making progress on 8586 so I'll just wait. My reminder entry is tireless in bringing up that I should look at this JIRA > Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients > configurable by a system property > - > > Key: SOLR-8500 > URL: https://issues.apache.org/jira/browse/SOLR-8500 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-8500.patch > > > Despite the warning in that code, in extremely high throughput situations > where there are guaranteed to be no updates to existing documents, it can be > useful to have more than one runner. > I envision this as an "expert" kind of thing, used only in situations where > the a-priori knowledge is that there are no updates to existing documents. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8500) Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients configurable by a system property
[ https://issues.apache.org/jira/browse/SOLR-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120410#comment-15120410 ] Mark Miller commented on SOLR-8500: --- The system as is simply can't correctly deal with these kind of reorders. I wish it wasn't true, but I wish I had a pony too :) > Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients > configurable by a system property > - > > Key: SOLR-8500 > URL: https://issues.apache.org/jira/browse/SOLR-8500 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-8500.patch > > > Despite the warning in that code, in extremely high throughput situations > where there are guaranteed to be no updates to existing documents, it can be > useful to have more than one runner. > I envision this as an "expert" kind of thing, used only in situations where > the a-priori knowledge is that there are no updates to existing documents. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8500) Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients configurable by a system property
[ https://issues.apache.org/jira/browse/SOLR-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120416#comment-15120416 ] Mark Miller commented on SOLR-8500: --- As far as a special case, the only special case that gets around this is if they never have to recover. > Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients > configurable by a system property > - > > Key: SOLR-8500 > URL: https://issues.apache.org/jira/browse/SOLR-8500 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-8500.patch > > > Despite the warning in that code, in extremely high throughput situations > where there are guaranteed to be no updates to existing documents, it can be > useful to have more than one runner. > I envision this as an "expert" kind of thing, used only in situations where > the a-priori knowledge is that there are no updates to existing documents. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8500) Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients configurable by a system property
[ https://issues.apache.org/jira/browse/SOLR-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119930#comment-15119930 ] Mark Miller commented on SOLR-8500: --- No, I don't think we should allow config to what we know will break the system, whether it's fat or not. If correctness does not matter, we can make things really fast. Once Yonik finishes the peer sync finger print it should no longer be a correctness issue to have these reorders though. > Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients > configurable by a system property > - > > Key: SOLR-8500 > URL: https://issues.apache.org/jira/browse/SOLR-8500 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-8500.patch > > > Despite the warning in that code, in extremely high throughput situations > where there are guaranteed to be no updates to existing documents, it can be > useful to have more than one runner. > I envision this as an "expert" kind of thing, used only in situations where > the a-priori knowledge is that there are no updates to existing documents. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8500) Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients configurable by a system property
[ https://issues.apache.org/jira/browse/SOLR-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119925#comment-15119925 ] Erick Erickson commented on SOLR-8500: -- [~markrmil...@gmail.com] So do you oppose this as a "use at your own risk under very special circumstances" kind of thing? This isn't theoretical, there are clients who have used a patch in here and are seeing significant benefits. > Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients > configurable by a system property > - > > Key: SOLR-8500 > URL: https://issues.apache.org/jira/browse/SOLR-8500 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-8500.patch > > > Despite the warning in that code, in extremely high throughput situations > where there are guaranteed to be no updates to existing documents, it can be > useful to have more than one runner. > I envision this as an "expert" kind of thing, used only in situations where > the a-priori knowledge is that there are no updates to existing documents. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8500) Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients configurable by a system property
[ https://issues.apache.org/jira/browse/SOLR-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15119939#comment-15119939 ] Mark Miller commented on SOLR-8500: --- In the short term, you can spin up more threads from the client rather than spinning up more threads here. > Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients > configurable by a system property > - > > Key: SOLR-8500 > URL: https://issues.apache.org/jira/browse/SOLR-8500 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-8500.patch > > > Despite the warning in that code, in extremely high throughput situations > where there are guaranteed to be no updates to existing documents, it can be > useful to have more than one runner. > I envision this as an "expert" kind of thing, used only in situations where > the a-priori knowledge is that there are no updates to existing documents. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8500) Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients configurable by a system property
[ https://issues.apache.org/jira/browse/SOLR-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120152#comment-15120152 ] Erick Erickson commented on SOLR-8500: -- First let me say I have only the most cursory understanding of "the reordering problem". My assumption is that since CUSC is batching up sub-lists of the update set and sending them in parallel that if doc1 is followed by doc2 in the original list, doc2 might get to the indexing node before doc1, be it an update, delete, add, whatever. That said, I don't really understand how reordering matters if (as per the original problem statement), it's _guaranteed_ that each document is new and is submitted exactly once _ever_. I guess another important restriction is that the client doesn't care if docs get into the index in a different order than they were sent. How would correctness be threatened in that situation? If the concern is that this is a too-specialized use-case that allows people to set it and shoot themselves in the foot too easily, that's a point. I just don't get why, in this specific use-case, this is a correctness question. All that said, if Yonik's fingerprint stuff is going in relatively soon, it's probably all moot and we can just wait on this... > Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients > configurable by a system property > - > > Key: SOLR-8500 > URL: https://issues.apache.org/jira/browse/SOLR-8500 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Minor > Attachments: SOLR-8500.patch > > > Despite the warning in that code, in extremely high throughput situations > where there are guaranteed to be no updates to existing documents, it can be > useful to have more than one runner. > I envision this as an "expert" kind of thing, used only in situations where > the a-priori knowledge is that there are no updates to existing documents. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8500) Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients configurable by a system property
[ https://issues.apache.org/jira/browse/SOLR-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15106132#comment-15106132 ] Erick Erickson commented on SOLR-8500: -- bq: I think using more than 1 thread may actually introduce more reordering problems right now. Does it matter in the case that I outlined? That there are no updates to existing documents to contend with so even if docs get reordered it shouldn't have any effects noticeably by the end user. Or am I missing the boat? > Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients > configurable by a system property > - > > Key: SOLR-8500 > URL: https://issues.apache.org/jira/browse/SOLR-8500 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Priority: Minor > Attachments: SOLR-8500.patch > > > Despite the warning in that code, in extremely high throughput situations > where there are guaranteed to be no updates to existing documents, it can be > useful to have more than one runner. > I envision this as an "expert" kind of thing, used only in situations where > the a-priori knowledge is that there are no updates to existing documents. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8500) Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients configurable by a system property
[ https://issues.apache.org/jira/browse/SOLR-8500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15090169#comment-15090169 ] Mark Miller commented on SOLR-8500: --- I think using more than 1 thread may actually introduce more reordering problems right now. > Allow the number of threads ConcurrentUpdateSolrClient StreamingSolrClients > configurable by a system property > - > > Key: SOLR-8500 > URL: https://issues.apache.org/jira/browse/SOLR-8500 > Project: Solr > Issue Type: Improvement >Reporter: Erick Erickson >Priority: Minor > Attachments: SOLR-8500.patch > > > Despite the warning in that code, in extremely high throughput situations > where there are guaranteed to be no updates to existing documents, it can be > useful to have more than one runner. > I envision this as an "expert" kind of thing, used only in situations where > the a-priori knowledge is that there are no updates to existing documents. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
