Re: Is it possible to find a leader from a list of cores in solr via java code
Hi, I got the solution to the above problem . Sharing the code so that it could help people in future PoolingClientConnectionManager poolingClientConnectionManager = new PoolingClientConnectionManager(); poolingClientConnectionManager.setMaxTotal(2); poolingClientConnectionManager.setDefaultMaxPerRoute(1); HttpClient httpClient = (HttpClient)new DefaultHttpClient(poolingClientConnectionManager); LBHttpSolrServer lbServer = new LBHttpSolrServer(httpClient); server = new CloudSolrServer(zkhost, lbServer); server.setDefaultCollection(collectionName); Thanks a lot to every1 in the thread chain. Your suggestions helped a lot -- View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4078012.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Is it possible to find a leader from a list of cores in solr via java code
Hi, As per the suggestions above I shifted my focus to using CloudSolrServer. In terms of sending updates to the leaders and reducing network traffic it works great. But i faced one problem in using CloudSolrServer is that it opens too many connections as large as five thousand connections. My Code is as follows ModifiableSolrParams params = new ModifiableSolrParams(); params.set(HttpClientUtil.PROP_MAX_CONNECTIONS, 3); params.set(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 2); HttpClient client = HttpClientUtil.createClient(params); LBHttpSolrServer lbServer = new LBHttpSolrServer(client); server = new CloudSolrServer(zkHost,lbServer); server.setDefaultCollection(defaultColllection); If there is only one instance of solr up then this works great. But in 1 shard 1 replica system it opens up too many connections in waiting state. Am I doing something incorrect. Any help would be highly appreciated -- View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4077587.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Is it possible to find a leader from a list of cores in solr via java code
Update requests are routed only to leader nodes by CloudSolrServer so there is a 1/numShards probability of avoiding an extra network hop. It is *not* a 1/(number of nodes in cluster) probability as you may think. You may also be interested in https://issues.apache.org/jira/browse/SOLR-4816 . Perhaps you can try the patch and report your results. On Sun, Jul 7, 2013 at 11:18 AM, Jack Krupansky j...@basetechnology.com wrote: There are three concepts to grasp: 1. You can send Solr update requests to ANY node of the cluster. Period. 2. Any extra network traffic (within the cluster) is likely to be negligible and absolutely not worrying about unless you have definitive evidence to the contrary. 3. Leader nodes in SolrCloud are designed to be dynamic and will change over time. Generally, you won't know which node is the leader for a shard at any given moment - it may have been a particular node one milliscond ago, but now maybe another node has been designated as the leader. Neither the application nor the user designates which node is a leader for a shard - it is a dynamic election process. Sure, you can check to see which node is currently the leader, but one millisecond later some other node may have gotten elected to be the leader. To repeat, you DO NOT need to be concerned with sending Solr update requests to a shard leader. The SolrCloud concept of leader is really only needed WITHIN the cluster. Clients outside of the SolrCloud cluster need not concern themselves with shard leaders. Micro-optimization and premature optimization are very poor tools to lead a system design. Focus the energy on the data modeling and the overall application design. -- Jack Krupansky -Original Message- From: vicky desai Sent: Sunday, July 07, 2013 1:17 AM To: solr-user@lucene.apache.org Subject: Re: Is it possible to find a leader from a list of cores in solr via java code Hi Erik, I just wanted to clarify if u got my concern right. If i send some documents to the replica core wont it first have to send the documents to the leader core which in turn would be sending it back to the replica cores. If yes then this will lead to additional network traffic which can be avoided by sending the documents directly to leader. Please correct me if I have got the concept incorrect.Any help is appreciated Thanks, Vicky -- View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4076016.html Sent from the Solr - User mailing list archive at Nabble.com. -- Regards, Shalin Shekhar Mangar.
Re: Is it possible to find a leader from a list of cores in solr via java code
What Jack and Shalin said Jack's comments about leaders changing is absolutely correct, although practically they won't change all that often unless you're bouncing nodes all over the place. But as Shalin says, CloudSolrServer will already to this, at least as far as sending updates to a leader. If you only have one shard, it's already been done for you, just use SolrJ and CloudSolrServer. And https://issues.apache.org/jira/browse/SOLR-4816 will further refine the process such that the documents are split up into packets, all the docs for that packet go to the same shard, and the packet will be sent to the shard leader. So as Jack says, this is probably a really poor place to put _your_ development efforts since a in the single-shard case it's already been done for you and b in the near future it'll already have been done for you in the multi-shard case. If you want to pursue this, I suggest putting your efforts into helping test SOLR-4816 is both more efficient for you and a help to the larger community. Best Erick On Sun, Jul 7, 2013 at 2:52 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: Update requests are routed only to leader nodes by CloudSolrServer so there is a 1/numShards probability of avoiding an extra network hop. It is *not* a 1/(number of nodes in cluster) probability as you may think. You may also be interested in https://issues.apache.org/jira/browse/SOLR-4816 . Perhaps you can try the patch and report your results. On Sun, Jul 7, 2013 at 11:18 AM, Jack Krupansky j...@basetechnology.com wrote: There are three concepts to grasp: 1. You can send Solr update requests to ANY node of the cluster. Period. 2. Any extra network traffic (within the cluster) is likely to be negligible and absolutely not worrying about unless you have definitive evidence to the contrary. 3. Leader nodes in SolrCloud are designed to be dynamic and will change over time. Generally, you won't know which node is the leader for a shard at any given moment - it may have been a particular node one milliscond ago, but now maybe another node has been designated as the leader. Neither the application nor the user designates which node is a leader for a shard - it is a dynamic election process. Sure, you can check to see which node is currently the leader, but one millisecond later some other node may have gotten elected to be the leader. To repeat, you DO NOT need to be concerned with sending Solr update requests to a shard leader. The SolrCloud concept of leader is really only needed WITHIN the cluster. Clients outside of the SolrCloud cluster need not concern themselves with shard leaders. Micro-optimization and premature optimization are very poor tools to lead a system design. Focus the energy on the data modeling and the overall application design. -- Jack Krupansky -Original Message- From: vicky desai Sent: Sunday, July 07, 2013 1:17 AM To: solr-user@lucene.apache.org Subject: Re: Is it possible to find a leader from a list of cores in solr via java code Hi Erik, I just wanted to clarify if u got my concern right. If i send some documents to the replica core wont it first have to send the documents to the leader core which in turn would be sending it back to the replica cores. If yes then this will lead to additional network traffic which can be avoided by sending the documents directly to leader. Please correct me if I have got the concept incorrect.Any help is appreciated Thanks, Vicky -- View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4076016.html Sent from the Solr - User mailing list archive at Nabble.com. -- Regards, Shalin Shekhar Mangar.
Re: Is it possible to find a leader from a list of cores in solr via java code
bq: Reason being If a write request is sent to the replica it relays it to the leader and then the leader relays it to all the replicas. This will help me in saving some network traffic as my application performs continuous writes How are you indexing? SolrJ already is leader aware and sends updates to the leaders. Requests should go to any node, there's no good reason to send requests only to replicas since the replicas and leaders are both search and index nodes. And an upcoming enhancement to SolrJ will additionally route the documents to the correct leader, i.e. it'll divide documents up into packets that all live on the same shard and send those packets to the leader for that shard. Unless you can prove that this is actually causing you trouble, I'd advise not spending the effort on this and focusing on other aspects of your project.. This _really_ feels like premature optimization. Best Erick On Wed, Jul 3, 2013 at 8:27 AM, vicky desai vicky.de...@germinait.comwrote: Hi, I have a requirement where in I want to write to the leader and read from the replica. Reason being If a write request is sent to the replica it relays it to the leader and then the leader relays it to all the replicas. This will help me in saving some network traffic as my application performs continuous writes -- View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4075083.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Is it possible to find a leader from a list of cores in solr via java code
Hi Erik, I just wanted to clarify if u got my concern right. If i send some documents to the replica core wont it first have to send the documents to the leader core which in turn would be sending it back to the replica cores. If yes then this will lead to additional network traffic which can be avoided by sending the documents directly to leader. Please correct me if I have got the concept incorrect.Any help is appreciated Thanks, Vicky -- View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4076016.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Is it possible to find a leader from a list of cores in solr via java code
There are three concepts to grasp: 1. You can send Solr update requests to ANY node of the cluster. Period. 2. Any extra network traffic (within the cluster) is likely to be negligible and absolutely not worrying about unless you have definitive evidence to the contrary. 3. Leader nodes in SolrCloud are designed to be dynamic and will change over time. Generally, you won't know which node is the leader for a shard at any given moment - it may have been a particular node one milliscond ago, but now maybe another node has been designated as the leader. Neither the application nor the user designates which node is a leader for a shard - it is a dynamic election process. Sure, you can check to see which node is currently the leader, but one millisecond later some other node may have gotten elected to be the leader. To repeat, you DO NOT need to be concerned with sending Solr update requests to a shard leader. The SolrCloud concept of leader is really only needed WITHIN the cluster. Clients outside of the SolrCloud cluster need not concern themselves with shard leaders. Micro-optimization and premature optimization are very poor tools to lead a system design. Focus the energy on the data modeling and the overall application design. -- Jack Krupansky -Original Message- From: vicky desai Sent: Sunday, July 07, 2013 1:17 AM To: solr-user@lucene.apache.org Subject: Re: Is it possible to find a leader from a list of cores in solr via java code Hi Erik, I just wanted to clarify if u got my concern right. If i send some documents to the replica core wont it first have to send the documents to the leader core which in turn would be sending it back to the replica cores. If yes then this will lead to additional network traffic which can be avoided by sending the documents directly to leader. Please correct me if I have got the concept incorrect.Any help is appreciated Thanks, Vicky -- View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4076016.html Sent from the Solr - User mailing list archive at Nabble.com.
Is it possible to find a leader from a list of cores in solr via java code
Hi , I have a set up of 1 leader and 1 replica and I have a requirement where in I need to find the leader core from the collection. Is there an api in solrj by means of which this can be achieved. -- View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Is it possible to find a leader from a list of cores in solr via java code
You can always query Zookeeper and find that information out. Take a look at CloudSolrServer, maybe ZkCoreNodeProps etc. for examples since CloudSolrServer is leader aware, it should have some clues... Or maybe ZkStateReader? I haven't been in that code much, so I can't be more specific... But why do you have this requirement? What do you hope to accomplish? Because this is often the kind of thing that's seems more useful than it is... Best Erick On Wed, Jul 3, 2013 at 3:05 AM, vicky desai vicky.de...@germinait.comwrote: Hi , I have a set up of 1 leader and 1 replica and I have a requirement where in I need to find the leader core from the collection. Is there an api in solrj by means of which this can be achieved. -- View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Is it possible to find a leader from a list of cores in solr via java code
I have the same question. My purpose is to start the dih full process on the leader and not on a replica. I tried full import on a replica but watching logs it seemed to me that the replica was loading data to send it to the leader which in turn has to update all the replicas. At least this is what I saw with solr 4.2.1 Giovanni 2013/7/3 Erick Erickson erickerick...@gmail.com You can always query Zookeeper and find that information out. Take a look at CloudSolrServer, maybe ZkCoreNodeProps etc. for examples since CloudSolrServer is leader aware, it should have some clues... Or maybe ZkStateReader? I haven't been in that code much, so I can't be more specific... But why do you have this requirement? What do you hope to accomplish? Because this is often the kind of thing that's seems more useful than it is... Best Erick On Wed, Jul 3, 2013 at 3:05 AM, vicky desai vicky.de...@germinait.com wrote: Hi , I have a set up of 1 leader and 1 replica and I have a requirement where in I need to find the leader core from the collection. Is there an api in solrj by means of which this can be achieved. -- View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Is it possible to find a leader from a list of cores in solr via java code
Hi, I have a requirement where in I want to write to the leader and read from the replica. Reason being If a write request is sent to the replica it relays it to the leader and then the leader relays it to all the replicas. This will help me in saving some network traffic as my application performs continuous writes -- View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4075083.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Is it possible to find a leader from a list of cores in solr via java code
One option could be to get the clusterstate.json via the following Solr url figure out the leader from the response json: * http://server:port/solr/zookeeper?detail=truepath=%2Fclusterstate.json* On Wed, Jul 3, 2013 at 5:57 PM, vicky desai vicky.de...@germinait.comwrote: Hi, I have a requirement where in I want to write to the leader and read from the replica. Reason being If a write request is sent to the replica it relays it to the leader and then the leader relays it to all the replicas. This will help me in saving some network traffic as my application performs continuous writes -- View this message in context: http://lucene.472066.n3.nabble.com/Is-it-possible-to-find-a-leader-from-a-list-of-cores-in-solr-via-java-code-tp4074994p4075083.html Sent from the Solr - User mailing list archive at Nabble.com.
