Re: Parallel Updates

2016-04-04 Thread Shawn Heisey
On 4/4/2016 3:46 PM, Robert Brown wrote: > I have 2 shards, 1 replica in each. > > The issue is the external processing job(s) I have to convert external > data into JSON, and then upload it via cURL. > > Will one Solr server only accept one update at a time and have any > others queued? (And poss

Re: Parallel Updates

2016-04-04 Thread Anshum Gupta
Solr would push all updates to all shards that are supposed to host the data. The documents are initially forwarded to the leader of the shard, which can dynamically change and the leader is responsible for versioning and ensuring replication across the followers but other than that, all nodes woul

Re: Parallel Updates

2016-04-04 Thread John Bickerstaff
Does SOLR cloud push indexing across all nodes? I've been planning 4 SOLR boxes with only 3 exposed via the load balancer, leaving the 4th available internally for my microservices to hit with indexing work. I was assuming that if I hit my "solr4" IP address, only "solr4" will do the indexing...

Re: Parallel Updates

2016-04-04 Thread Anshum Gupta
The short answer is - There's no real limit on Solr in terms of concurrency. Here are a few things that would impact your numbers though: * What version of Solr are you using and how ? i.e. SolrCloud, standalone, traditional replication ? * Do you use atomic updates? * How do you index ? Assuming

Re: Parallel Updates

2016-04-04 Thread Robert Brown
Thanks John, I have 2 shards, 1 replica in each. The issue is the external processing job(s) I have to convert external data into JSON, and then upload it via cURL. Will one Solr server only accept one update at a time and have any others queued? (And possibly timeout). I like the idea of

Re: Parallel Updates

2016-04-04 Thread John Bickerstaff
Will the processes be Solr processes? Or do you mean multiple threads hitting the same Solr server(s)? There will be a natural bottleneck at one Solr server if you are hitting it with a lot of threads - since that one server will have to do all the indexing. I don't know if this idea is helpful,

Parallel Updates

2016-04-04 Thread Robert Brown
Hi, Does Solr have any sort of limit when attempting multiple updates, from separate clients? Are there any safe thresholds one should try to stay within? I have an index of around 60m documents that gets updated at key points during the day from ~200 downloaded files - I'd like to fork off