RE: Is it possible to manually select a shard leader in a running SolrCloud?
Thanks Mark, Sounds good. We are still at the load test stage and will see how this goes. I imagine this is more of a concerning in concept than in reality. :) Cheers, Tim -Original Message- From: Mark Miller [mailto:markrmil...@gmail.com] Sent: Thursday, February 21, 2013 7:52 PM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? The leader doesn't really do a lot more work than any of the replicas, so I don't think it's likely that important. If someone starts running into problems, that's usually when we start looking for solutions. - Mark On Feb 21, 2013, at 10:20 PM, Vaillancourt, Tim tvaillanco...@ea.com wrote: I sent this request to ServerA in this case, which became the leader of all shards. As far as I know you're supposed to issue this call to just one server as it issues the calls to the other leaders/replicas in the background, right? I am expecting the single collections API call to spread the leaders evenly across SOLR instances. Hopefully I am just doing/expecting something wrong :). Tim Vaillancourt -Original Message- From: Upayavira [mailto:u...@odoko.co.uk] Sent: Thursday, February 21, 2013 1:44 PM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? Which of your three hosts did you point this request at? Upayavira On Thu, Feb 21, 2013, at 09:13 PM, Vaillancourt, Tim wrote: Correction, I used this curl: curl -v 'http://HOST:8983/solr/admin/collections?action=CREATEname=testnumShards=3replicationFactor=2maxShardsPerNode=2' So 3 instances, 3 shards, 2 replicas per shard. ServerA becomes leader of all 3 shards in 4.1 with this call. Tim Vaillancourt -Original Message- From: Vaillancourt, Tim [mailto:tvaillanco...@ea.com] Sent: Thursday, February 21, 2013 11:27 AM To: solr-user@lucene.apache.org; markrmil...@gmail.com Subject: RE: Is it possible to manually select a shard leader in a running SolrCloud? Thanks Mark, The real driver for me wanting to promote a different leader is when I create a new Collection via the Collections API across a multi-server SolrCloud, the leader of each shard is always the same host, so you're right that I'm tackling the wrong problem with this request, although it would fix it for me. If I create the cores manually via the cores API, one-by-one, I am able to get what I expect, but when running this Collections API call on a 3 SOLR 4.1 instance, 3 shard setup, 1 server becomes the leader of all 3 shards, meaning it will get all the writes for everything (correct me if I am wrong). If so, this will not scale well with all writes to one node (or correct me if I am wrong)? curl -v 'http://HOST:8983/solr/admin/collections?action=CREATEname=testnumShards=3replicationFactor=1maxShardsPerNode=2' Currently on my 3 instance SOLR 4.1 setup, the above call creates the following: - ServerA is the leader of all 3 shards (the problem I want to address). - ServerB + ServerC are automagically replicas of the 3 leader shards on ServerA. So again, my issue is one server gets all the writes. Does anyone else encounter this? If so, I should spawn a separate thread on my specific issue. Cheers, Tim -Original Message- From: Mark Miller [mailto:markrmil...@gmail.com] Sent: Tuesday, February 19, 2013 8:44 PM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? You can't easily do it the way it's implemented in ZooKeeper. We would probably internally have to do the same thing - elect a new leader and drop him until the one we wanted came up. The main thing doing it internally would gain is that you could skip the elected guy from becoming the actual leader and just move on to the next candidate. Still some tricky corner cases to deal with and such as well. I think for most things you would use this to solve, there is probably an alternate thing that should be addressed. - Mark On Mon, Feb 18, 2013 at 4:15 PM, Vaillancourt, Tim tvaillanco...@ea.com wrote: Hey all, I feel having to unload the leader core to force an election is hacky, and as far as I know would still leave which node becomes the Leader to chance, ie I cannot guarantee NodeX becomes Leader 100% in all cases. Also, this imposes additional load temporarily. Is there a way to force the winner of the Election, and if not, is there a known feature-request for this? Cheers, Tim Vaillancourt -Original Message- From: Joseph Dale [mailto:joey.d...@gmail.com] Sent: Sunday, February 03, 2013 7:42 AM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? With solrclound all cores are collections. The collections API it just a wrapper to call the
Re: Is it possible to manually select a shard leader in a running SolrCloud?
i think the use case here is more of a management one. -wanting to explicitly configure a specific node as leader (the reasons for this could vary) -wanting to gracefully/safely move a leader role from a specific node without going thru an actual election process (as was mentioned previously, why introduce variability?)..some sanity checks ala role moves if nodes are actually in sync, etc. otherwise it stays put the most obvious situation where i could see this being beneficial would be for minimizing elections, and the transitions, when doing wide service restarts for config changes, etc (restart all the replicas, gracefully move the leader to a restarted replica cleanly and orderly once everything is back in sync, restart the now former leader) On Thu, Feb 21, 2013 at 7:51 PM, Mark Miller markrmil...@gmail.com wrote: The leader doesn't really do a lot more work than any of the replicas, so I don't think it's likely that important. If someone starts running into problems, that's usually when we start looking for solutions. - Mark On Feb 21, 2013, at 10:20 PM, Vaillancourt, Tim tvaillanco...@ea.com wrote: I sent this request to ServerA in this case, which became the leader of all shards. As far as I know you're supposed to issue this call to just one server as it issues the calls to the other leaders/replicas in the background, right? I am expecting the single collections API call to spread the leaders evenly across SOLR instances. Hopefully I am just doing/expecting something wrong :). Tim Vaillancourt -Original Message- From: Upayavira [mailto:u...@odoko.co.uk] Sent: Thursday, February 21, 2013 1:44 PM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? Which of your three hosts did you point this request at? Upayavira On Thu, Feb 21, 2013, at 09:13 PM, Vaillancourt, Tim wrote: Correction, I used this curl: curl -v 'http://HOST:8983/solr/admin/collections?action=CREATEname=testnumShards=3replicationFactor=2maxShardsPerNode=2' So 3 instances, 3 shards, 2 replicas per shard. ServerA becomes leader of all 3 shards in 4.1 with this call. Tim Vaillancourt -Original Message- From: Vaillancourt, Tim [mailto:tvaillanco...@ea.com] Sent: Thursday, February 21, 2013 11:27 AM To: solr-user@lucene.apache.org; markrmil...@gmail.com Subject: RE: Is it possible to manually select a shard leader in a running SolrCloud? Thanks Mark, The real driver for me wanting to promote a different leader is when I create a new Collection via the Collections API across a multi-server SolrCloud, the leader of each shard is always the same host, so you're right that I'm tackling the wrong problem with this request, although it would fix it for me. If I create the cores manually via the cores API, one-by-one, I am able to get what I expect, but when running this Collections API call on a 3 SOLR 4.1 instance, 3 shard setup, 1 server becomes the leader of all 3 shards, meaning it will get all the writes for everything (correct me if I am wrong). If so, this will not scale well with all writes to one node (or correct me if I am wrong)? curl -v 'http://HOST:8983/solr/admin/collections?action=CREATEname=testnumShards=3replicationFactor=1maxShardsPerNode=2' Currently on my 3 instance SOLR 4.1 setup, the above call creates the following: - ServerA is the leader of all 3 shards (the problem I want to address). - ServerB + ServerC are automagically replicas of the 3 leader shards on ServerA. So again, my issue is one server gets all the writes. Does anyone else encounter this? If so, I should spawn a separate thread on my specific issue. Cheers, Tim -Original Message- From: Mark Miller [mailto:markrmil...@gmail.com] Sent: Tuesday, February 19, 2013 8:44 PM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? You can't easily do it the way it's implemented in ZooKeeper. We would probably internally have to do the same thing - elect a new leader and drop him until the one we wanted came up. The main thing doing it internally would gain is that you could skip the elected guy from becoming the actual leader and just move on to the next candidate. Still some tricky corner cases to deal with and such as well. I think for most things you would use this to solve, there is probably an alternate thing that should be addressed. - Mark On Mon, Feb 18, 2013 at 4:15 PM, Vaillancourt, Tim tvaillanco...@ea.com wrote: Hey all, I feel having to unload the leader core to force an election is hacky, and as far as I know would still leave which node becomes the Leader to chance, ie I cannot guarantee NodeX becomes Leader 100% in all cases. Also, this imposes additional load temporarily. Is there a way to force the winner of the Election, and if not, is there a known feature-request for
RE: Is it possible to manually select a shard leader in a running SolrCloud?
Yeah, exactly - although there are workarounds, this is probably worth a feature request whether or not it is turned down. I have created these 2 new feature requests (after not being able to find duplicates through searching): 1) Please add support for manual leader election/promotion: https://issues.apache.org/jira/browse/SOLR-4491 2) Please add support for Collection API CREATE method to evenly distribute leader roles among instances: https://issues.apache.org/jira/browse/SOLR-4492 Thanks Mark/all! Tim Vaillancourt -Original Message- From: Boogie Shafer [mailto:boo...@ebrary.com] Sent: Friday, February 22, 2013 12:05 PM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? i think the use case here is more of a management one. -wanting to explicitly configure a specific node as leader (the reasons for this could vary) -wanting to gracefully/safely move a leader role from a specific node without going thru an actual election process (as was mentioned previously, why introduce variability?)..some sanity checks ala role moves if nodes are actually in sync, etc. otherwise it stays put the most obvious situation where i could see this being beneficial would be for minimizing elections, and the transitions, when doing wide service restarts for config changes, etc (restart all the replicas, gracefully move the leader to a restarted replica cleanly and orderly once everything is back in sync, restart the now former leader) On Thu, Feb 21, 2013 at 7:51 PM, Mark Miller markrmil...@gmail.com wrote: The leader doesn't really do a lot more work than any of the replicas, so I don't think it's likely that important. If someone starts running into problems, that's usually when we start looking for solutions. - Mark On Feb 21, 2013, at 10:20 PM, Vaillancourt, Tim tvaillanco...@ea.com wrote: I sent this request to ServerA in this case, which became the leader of all shards. As far as I know you're supposed to issue this call to just one server as it issues the calls to the other leaders/replicas in the background, right? I am expecting the single collections API call to spread the leaders evenly across SOLR instances. Hopefully I am just doing/expecting something wrong :). Tim Vaillancourt -Original Message- From: Upayavira [mailto:u...@odoko.co.uk] Sent: Thursday, February 21, 2013 1:44 PM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? Which of your three hosts did you point this request at? Upayavira On Thu, Feb 21, 2013, at 09:13 PM, Vaillancourt, Tim wrote: Correction, I used this curl: curl -v 'http://HOST:8983/solr/admin/collections?action=CREATEname=testnumShards=3replicationFactor=2maxShardsPerNode=2' So 3 instances, 3 shards, 2 replicas per shard. ServerA becomes leader of all 3 shards in 4.1 with this call. Tim Vaillancourt -Original Message- From: Vaillancourt, Tim [mailto:tvaillanco...@ea.com] Sent: Thursday, February 21, 2013 11:27 AM To: solr-user@lucene.apache.org; markrmil...@gmail.com Subject: RE: Is it possible to manually select a shard leader in a running SolrCloud? Thanks Mark, The real driver for me wanting to promote a different leader is when I create a new Collection via the Collections API across a multi-server SolrCloud, the leader of each shard is always the same host, so you're right that I'm tackling the wrong problem with this request, although it would fix it for me. If I create the cores manually via the cores API, one-by-one, I am able to get what I expect, but when running this Collections API call on a 3 SOLR 4.1 instance, 3 shard setup, 1 server becomes the leader of all 3 shards, meaning it will get all the writes for everything (correct me if I am wrong). If so, this will not scale well with all writes to one node (or correct me if I am wrong)? curl -v 'http://HOST:8983/solr/admin/collections?action=CREATEname=testnumShards=3replicationFactor=1maxShardsPerNode=2' Currently on my 3 instance SOLR 4.1 setup, the above call creates the following: - ServerA is the leader of all 3 shards (the problem I want to address). - ServerB + ServerC are automagically replicas of the 3 leader shards on ServerA. So again, my issue is one server gets all the writes. Does anyone else encounter this? If so, I should spawn a separate thread on my specific issue. Cheers, Tim -Original Message- From: Mark Miller [mailto:markrmil...@gmail.com] Sent: Tuesday, February 19, 2013 8:44 PM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? You can't easily do it the way it's implemented in ZooKeeper. We would probably internally have to do the same thing - elect a new leader and drop him until the one we wanted came up. The main thing
RE: Is it possible to manually select a shard leader in a running SolrCloud?
Thanks Mark, The real driver for me wanting to promote a different leader is when I create a new Collection via the Collections API across a multi-server SolrCloud, the leader of each shard is always the same host, so you're right that I'm tackling the wrong problem with this request, although it would fix it for me. If I create the cores manually via the cores API, one-by-one, I am able to get what I expect, but when running this Collections API call on a 3 SOLR 4.1 instance, 3 shard setup, 1 server becomes the leader of all 3 shards, meaning it will get all the writes for everything (correct me if I am wrong). If so, this will not scale well with all writes to one node (or correct me if I am wrong)? curl -v 'http://HOST:8983/solr/admin/collections?action=CREATEname=testnumShards=3replicationFactor=1maxShardsPerNode=2' Currently on my 3 instance SOLR 4.1 setup, the above call creates the following: - ServerA is the leader of all 3 shards (the problem I want to address). - ServerB + ServerC are automagically replicas of the 3 leader shards on ServerA. So again, my issue is one server gets all the writes. Does anyone else encounter this? If so, I should spawn a separate thread on my specific issue. Cheers, Tim -Original Message- From: Mark Miller [mailto:markrmil...@gmail.com] Sent: Tuesday, February 19, 2013 8:44 PM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? You can't easily do it the way it's implemented in ZooKeeper. We would probably internally have to do the same thing - elect a new leader and drop him until the one we wanted came up. The main thing doing it internally would gain is that you could skip the elected guy from becoming the actual leader and just move on to the next candidate. Still some tricky corner cases to deal with and such as well. I think for most things you would use this to solve, there is probably an alternate thing that should be addressed. - Mark On Mon, Feb 18, 2013 at 4:15 PM, Vaillancourt, Tim tvaillanco...@ea.com wrote: Hey all, I feel having to unload the leader core to force an election is hacky, and as far as I know would still leave which node becomes the Leader to chance, ie I cannot guarantee NodeX becomes Leader 100% in all cases. Also, this imposes additional load temporarily. Is there a way to force the winner of the Election, and if not, is there a known feature-request for this? Cheers, Tim Vaillancourt -Original Message- From: Joseph Dale [mailto:joey.d...@gmail.com] Sent: Sunday, February 03, 2013 7:42 AM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? With solrclound all cores are collections. The collections API it just a wrapper to call the core api a million times with one command. to /solr/admin/cores?action=CREATEname=core1collection=core1shard=1 Basically your creating the shard again, after leader props have gone out. Solr will check ZK and find a core meeting that description, then simply get a copy of the index from the leader of that shard. On Feb 3, 2013, at 10:37 AM, Brett Hoerner br...@bretthoerner.com wrote: What is the inverse I'd use to re-create/load a core on another machine but make sure it's also known to SolrCloud/as a shard? On Sat, Feb 2, 2013 at 4:01 PM, Joseph Dale joey.d...@gmail.com wrote: To be more clear lets say bob it the leader of core 1. On bob do a /admin/cores?action=unloadname=core1. This removes the core/shard from bob, giving the other servers a chance to grab leader props. -Joey On Feb 2, 2013, at 11:27 AM, Brett Hoerner br...@bretthoerner.com wrote: Hi, I have a 5 server cluster running 1 collection with 20 shards, replication factor of 2. Earlier this week I had to do a rolling restart across the cluster, this worked great and the cluster stayed up the whole time. The problem is that the last node I restarted is now the leader of 0 shards, and is just holding replicas. I've noticed this node has abnormally high load average, while the other nodes (who have the same number of shards, but more leaders on average) are fine. First, I'm wondering if that loud could be related to being a 5x replica and 0x leader? Second, I was wondering if I could somehow flag single shards to re-elect a leader (or force a leader) so that I could more evenly distribute how many leader shards each physical server has running? Thanks. -- - Mark
RE: Is it possible to manually select a shard leader in a running SolrCloud?
Correction, I used this curl: curl -v 'http://HOST:8983/solr/admin/collections?action=CREATEname=testnumShards=3replicationFactor=2maxShardsPerNode=2' So 3 instances, 3 shards, 2 replicas per shard. ServerA becomes leader of all 3 shards in 4.1 with this call. Tim Vaillancourt -Original Message- From: Vaillancourt, Tim [mailto:tvaillanco...@ea.com] Sent: Thursday, February 21, 2013 11:27 AM To: solr-user@lucene.apache.org; markrmil...@gmail.com Subject: RE: Is it possible to manually select a shard leader in a running SolrCloud? Thanks Mark, The real driver for me wanting to promote a different leader is when I create a new Collection via the Collections API across a multi-server SolrCloud, the leader of each shard is always the same host, so you're right that I'm tackling the wrong problem with this request, although it would fix it for me. If I create the cores manually via the cores API, one-by-one, I am able to get what I expect, but when running this Collections API call on a 3 SOLR 4.1 instance, 3 shard setup, 1 server becomes the leader of all 3 shards, meaning it will get all the writes for everything (correct me if I am wrong). If so, this will not scale well with all writes to one node (or correct me if I am wrong)? curl -v 'http://HOST:8983/solr/admin/collections?action=CREATEname=testnumShards=3replicationFactor=1maxShardsPerNode=2' Currently on my 3 instance SOLR 4.1 setup, the above call creates the following: - ServerA is the leader of all 3 shards (the problem I want to address). - ServerB + ServerC are automagically replicas of the 3 leader shards on ServerA. So again, my issue is one server gets all the writes. Does anyone else encounter this? If so, I should spawn a separate thread on my specific issue. Cheers, Tim -Original Message- From: Mark Miller [mailto:markrmil...@gmail.com] Sent: Tuesday, February 19, 2013 8:44 PM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? You can't easily do it the way it's implemented in ZooKeeper. We would probably internally have to do the same thing - elect a new leader and drop him until the one we wanted came up. The main thing doing it internally would gain is that you could skip the elected guy from becoming the actual leader and just move on to the next candidate. Still some tricky corner cases to deal with and such as well. I think for most things you would use this to solve, there is probably an alternate thing that should be addressed. - Mark On Mon, Feb 18, 2013 at 4:15 PM, Vaillancourt, Tim tvaillanco...@ea.com wrote: Hey all, I feel having to unload the leader core to force an election is hacky, and as far as I know would still leave which node becomes the Leader to chance, ie I cannot guarantee NodeX becomes Leader 100% in all cases. Also, this imposes additional load temporarily. Is there a way to force the winner of the Election, and if not, is there a known feature-request for this? Cheers, Tim Vaillancourt -Original Message- From: Joseph Dale [mailto:joey.d...@gmail.com] Sent: Sunday, February 03, 2013 7:42 AM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? With solrclound all cores are collections. The collections API it just a wrapper to call the core api a million times with one command. to /solr/admin/cores?action=CREATEname=core1collection=core1shard=1 Basically your creating the shard again, after leader props have gone out. Solr will check ZK and find a core meeting that description, then simply get a copy of the index from the leader of that shard. On Feb 3, 2013, at 10:37 AM, Brett Hoerner br...@bretthoerner.com wrote: What is the inverse I'd use to re-create/load a core on another machine but make sure it's also known to SolrCloud/as a shard? On Sat, Feb 2, 2013 at 4:01 PM, Joseph Dale joey.d...@gmail.com wrote: To be more clear lets say bob it the leader of core 1. On bob do a /admin/cores?action=unloadname=core1. This removes the core/shard from bob, giving the other servers a chance to grab leader props. -Joey On Feb 2, 2013, at 11:27 AM, Brett Hoerner br...@bretthoerner.com wrote: Hi, I have a 5 server cluster running 1 collection with 20 shards, replication factor of 2. Earlier this week I had to do a rolling restart across the cluster, this worked great and the cluster stayed up the whole time. The problem is that the last node I restarted is now the leader of 0 shards, and is just holding replicas. I've noticed this node has abnormally high load average, while the other nodes (who have the same number of shards, but more leaders on average) are fine. First, I'm wondering if that loud could be related to being a 5x replica and 0x leader? Second, I was wondering if I could somehow flag single shards to re-elect a leader (or force a
Re: Is it possible to manually select a shard leader in a running SolrCloud?
Which of your three hosts did you point this request at? Upayavira On Thu, Feb 21, 2013, at 09:13 PM, Vaillancourt, Tim wrote: Correction, I used this curl: curl -v 'http://HOST:8983/solr/admin/collections?action=CREATEname=testnumShards=3replicationFactor=2maxShardsPerNode=2' So 3 instances, 3 shards, 2 replicas per shard. ServerA becomes leader of all 3 shards in 4.1 with this call. Tim Vaillancourt -Original Message- From: Vaillancourt, Tim [mailto:tvaillanco...@ea.com] Sent: Thursday, February 21, 2013 11:27 AM To: solr-user@lucene.apache.org; markrmil...@gmail.com Subject: RE: Is it possible to manually select a shard leader in a running SolrCloud? Thanks Mark, The real driver for me wanting to promote a different leader is when I create a new Collection via the Collections API across a multi-server SolrCloud, the leader of each shard is always the same host, so you're right that I'm tackling the wrong problem with this request, although it would fix it for me. If I create the cores manually via the cores API, one-by-one, I am able to get what I expect, but when running this Collections API call on a 3 SOLR 4.1 instance, 3 shard setup, 1 server becomes the leader of all 3 shards, meaning it will get all the writes for everything (correct me if I am wrong). If so, this will not scale well with all writes to one node (or correct me if I am wrong)? curl -v 'http://HOST:8983/solr/admin/collections?action=CREATEname=testnumShards=3replicationFactor=1maxShardsPerNode=2' Currently on my 3 instance SOLR 4.1 setup, the above call creates the following: - ServerA is the leader of all 3 shards (the problem I want to address). - ServerB + ServerC are automagically replicas of the 3 leader shards on ServerA. So again, my issue is one server gets all the writes. Does anyone else encounter this? If so, I should spawn a separate thread on my specific issue. Cheers, Tim -Original Message- From: Mark Miller [mailto:markrmil...@gmail.com] Sent: Tuesday, February 19, 2013 8:44 PM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? You can't easily do it the way it's implemented in ZooKeeper. We would probably internally have to do the same thing - elect a new leader and drop him until the one we wanted came up. The main thing doing it internally would gain is that you could skip the elected guy from becoming the actual leader and just move on to the next candidate. Still some tricky corner cases to deal with and such as well. I think for most things you would use this to solve, there is probably an alternate thing that should be addressed. - Mark On Mon, Feb 18, 2013 at 4:15 PM, Vaillancourt, Tim tvaillanco...@ea.com wrote: Hey all, I feel having to unload the leader core to force an election is hacky, and as far as I know would still leave which node becomes the Leader to chance, ie I cannot guarantee NodeX becomes Leader 100% in all cases. Also, this imposes additional load temporarily. Is there a way to force the winner of the Election, and if not, is there a known feature-request for this? Cheers, Tim Vaillancourt -Original Message- From: Joseph Dale [mailto:joey.d...@gmail.com] Sent: Sunday, February 03, 2013 7:42 AM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? With solrclound all cores are collections. The collections API it just a wrapper to call the core api a million times with one command. to /solr/admin/cores?action=CREATEname=core1collection=core1shard=1 Basically your creating the shard again, after leader props have gone out. Solr will check ZK and find a core meeting that description, then simply get a copy of the index from the leader of that shard. On Feb 3, 2013, at 10:37 AM, Brett Hoerner br...@bretthoerner.com wrote: What is the inverse I'd use to re-create/load a core on another machine but make sure it's also known to SolrCloud/as a shard? On Sat, Feb 2, 2013 at 4:01 PM, Joseph Dale joey.d...@gmail.com wrote: To be more clear lets say bob it the leader of core 1. On bob do a /admin/cores?action=unloadname=core1. This removes the core/shard from bob, giving the other servers a chance to grab leader props. -Joey On Feb 2, 2013, at 11:27 AM, Brett Hoerner br...@bretthoerner.com wrote: Hi, I have a 5 server cluster running 1 collection with 20 shards, replication factor of 2. Earlier this week I had to do a rolling restart across the cluster, this worked great and the cluster stayed up the whole time. The problem is that the last node I restarted is now the leader of 0 shards, and is just holding replicas. I've noticed this node has abnormally high load average, while the other nodes (who have the same number of
RE: Is it possible to manually select a shard leader in a running SolrCloud?
I sent this request to ServerA in this case, which became the leader of all shards. As far as I know you're supposed to issue this call to just one server as it issues the calls to the other leaders/replicas in the background, right? I am expecting the single collections API call to spread the leaders evenly across SOLR instances. Hopefully I am just doing/expecting something wrong :). Tim Vaillancourt -Original Message- From: Upayavira [mailto:u...@odoko.co.uk] Sent: Thursday, February 21, 2013 1:44 PM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? Which of your three hosts did you point this request at? Upayavira On Thu, Feb 21, 2013, at 09:13 PM, Vaillancourt, Tim wrote: Correction, I used this curl: curl -v 'http://HOST:8983/solr/admin/collections?action=CREATEname=testnumShards=3replicationFactor=2maxShardsPerNode=2' So 3 instances, 3 shards, 2 replicas per shard. ServerA becomes leader of all 3 shards in 4.1 with this call. Tim Vaillancourt -Original Message- From: Vaillancourt, Tim [mailto:tvaillanco...@ea.com] Sent: Thursday, February 21, 2013 11:27 AM To: solr-user@lucene.apache.org; markrmil...@gmail.com Subject: RE: Is it possible to manually select a shard leader in a running SolrCloud? Thanks Mark, The real driver for me wanting to promote a different leader is when I create a new Collection via the Collections API across a multi-server SolrCloud, the leader of each shard is always the same host, so you're right that I'm tackling the wrong problem with this request, although it would fix it for me. If I create the cores manually via the cores API, one-by-one, I am able to get what I expect, but when running this Collections API call on a 3 SOLR 4.1 instance, 3 shard setup, 1 server becomes the leader of all 3 shards, meaning it will get all the writes for everything (correct me if I am wrong). If so, this will not scale well with all writes to one node (or correct me if I am wrong)? curl -v 'http://HOST:8983/solr/admin/collections?action=CREATEname=testnumShards=3replicationFactor=1maxShardsPerNode=2' Currently on my 3 instance SOLR 4.1 setup, the above call creates the following: - ServerA is the leader of all 3 shards (the problem I want to address). - ServerB + ServerC are automagically replicas of the 3 leader shards on ServerA. So again, my issue is one server gets all the writes. Does anyone else encounter this? If so, I should spawn a separate thread on my specific issue. Cheers, Tim -Original Message- From: Mark Miller [mailto:markrmil...@gmail.com] Sent: Tuesday, February 19, 2013 8:44 PM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? You can't easily do it the way it's implemented in ZooKeeper. We would probably internally have to do the same thing - elect a new leader and drop him until the one we wanted came up. The main thing doing it internally would gain is that you could skip the elected guy from becoming the actual leader and just move on to the next candidate. Still some tricky corner cases to deal with and such as well. I think for most things you would use this to solve, there is probably an alternate thing that should be addressed. - Mark On Mon, Feb 18, 2013 at 4:15 PM, Vaillancourt, Tim tvaillanco...@ea.com wrote: Hey all, I feel having to unload the leader core to force an election is hacky, and as far as I know would still leave which node becomes the Leader to chance, ie I cannot guarantee NodeX becomes Leader 100% in all cases. Also, this imposes additional load temporarily. Is there a way to force the winner of the Election, and if not, is there a known feature-request for this? Cheers, Tim Vaillancourt -Original Message- From: Joseph Dale [mailto:joey.d...@gmail.com] Sent: Sunday, February 03, 2013 7:42 AM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? With solrclound all cores are collections. The collections API it just a wrapper to call the core api a million times with one command. to /solr/admin/cores?action=CREATEname=core1collection=core1shard=1 Basically your creating the shard again, after leader props have gone out. Solr will check ZK and find a core meeting that description, then simply get a copy of the index from the leader of that shard. On Feb 3, 2013, at 10:37 AM, Brett Hoerner br...@bretthoerner.com wrote: What is the inverse I'd use to re-create/load a core on another machine but make sure it's also known to SolrCloud/as a shard? On Sat, Feb 2, 2013 at 4:01 PM, Joseph Dale joey.d...@gmail.com wrote: To be more clear lets say bob it the leader of core 1. On bob do a
Re: Is it possible to manually select a shard leader in a running SolrCloud?
The leader doesn't really do a lot more work than any of the replicas, so I don't think it's likely that important. If someone starts running into problems, that's usually when we start looking for solutions. - Mark On Feb 21, 2013, at 10:20 PM, Vaillancourt, Tim tvaillanco...@ea.com wrote: I sent this request to ServerA in this case, which became the leader of all shards. As far as I know you're supposed to issue this call to just one server as it issues the calls to the other leaders/replicas in the background, right? I am expecting the single collections API call to spread the leaders evenly across SOLR instances. Hopefully I am just doing/expecting something wrong :). Tim Vaillancourt -Original Message- From: Upayavira [mailto:u...@odoko.co.uk] Sent: Thursday, February 21, 2013 1:44 PM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? Which of your three hosts did you point this request at? Upayavira On Thu, Feb 21, 2013, at 09:13 PM, Vaillancourt, Tim wrote: Correction, I used this curl: curl -v 'http://HOST:8983/solr/admin/collections?action=CREATEname=testnumShards=3replicationFactor=2maxShardsPerNode=2' So 3 instances, 3 shards, 2 replicas per shard. ServerA becomes leader of all 3 shards in 4.1 with this call. Tim Vaillancourt -Original Message- From: Vaillancourt, Tim [mailto:tvaillanco...@ea.com] Sent: Thursday, February 21, 2013 11:27 AM To: solr-user@lucene.apache.org; markrmil...@gmail.com Subject: RE: Is it possible to manually select a shard leader in a running SolrCloud? Thanks Mark, The real driver for me wanting to promote a different leader is when I create a new Collection via the Collections API across a multi-server SolrCloud, the leader of each shard is always the same host, so you're right that I'm tackling the wrong problem with this request, although it would fix it for me. If I create the cores manually via the cores API, one-by-one, I am able to get what I expect, but when running this Collections API call on a 3 SOLR 4.1 instance, 3 shard setup, 1 server becomes the leader of all 3 shards, meaning it will get all the writes for everything (correct me if I am wrong). If so, this will not scale well with all writes to one node (or correct me if I am wrong)? curl -v 'http://HOST:8983/solr/admin/collections?action=CREATEname=testnumShards=3replicationFactor=1maxShardsPerNode=2' Currently on my 3 instance SOLR 4.1 setup, the above call creates the following: - ServerA is the leader of all 3 shards (the problem I want to address). - ServerB + ServerC are automagically replicas of the 3 leader shards on ServerA. So again, my issue is one server gets all the writes. Does anyone else encounter this? If so, I should spawn a separate thread on my specific issue. Cheers, Tim -Original Message- From: Mark Miller [mailto:markrmil...@gmail.com] Sent: Tuesday, February 19, 2013 8:44 PM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? You can't easily do it the way it's implemented in ZooKeeper. We would probably internally have to do the same thing - elect a new leader and drop him until the one we wanted came up. The main thing doing it internally would gain is that you could skip the elected guy from becoming the actual leader and just move on to the next candidate. Still some tricky corner cases to deal with and such as well. I think for most things you would use this to solve, there is probably an alternate thing that should be addressed. - Mark On Mon, Feb 18, 2013 at 4:15 PM, Vaillancourt, Tim tvaillanco...@ea.com wrote: Hey all, I feel having to unload the leader core to force an election is hacky, and as far as I know would still leave which node becomes the Leader to chance, ie I cannot guarantee NodeX becomes Leader 100% in all cases. Also, this imposes additional load temporarily. Is there a way to force the winner of the Election, and if not, is there a known feature-request for this? Cheers, Tim Vaillancourt -Original Message- From: Joseph Dale [mailto:joey.d...@gmail.com] Sent: Sunday, February 03, 2013 7:42 AM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? With solrclound all cores are collections. The collections API it just a wrapper to call the core api a million times with one command. to /solr/admin/cores?action=CREATEname=core1collection=core1shard=1 Basically your creating the shard again, after leader props have gone out. Solr will check ZK and find a core meeting that description, then simply get a copy of the index from the leader of that shard. On Feb 3, 2013, at 10:37 AM, Brett Hoerner br...@bretthoerner.com wrote: What is
Re: Is it possible to manually select a shard leader in a running SolrCloud?
You can't easily do it the way it's implemented in ZooKeeper. We would probably internally have to do the same thing - elect a new leader and drop him until the one we wanted came up. The main thing doing it internally would gain is that you could skip the elected guy from becoming the actual leader and just move on to the next candidate. Still some tricky corner cases to deal with and such as well. I think for most things you would use this to solve, there is probably an alternate thing that should be addressed. - Mark On Mon, Feb 18, 2013 at 4:15 PM, Vaillancourt, Tim tvaillanco...@ea.com wrote: Hey all, I feel having to unload the leader core to force an election is hacky, and as far as I know would still leave which node becomes the Leader to chance, ie I cannot guarantee NodeX becomes Leader 100% in all cases. Also, this imposes additional load temporarily. Is there a way to force the winner of the Election, and if not, is there a known feature-request for this? Cheers, Tim Vaillancourt -Original Message- From: Joseph Dale [mailto:joey.d...@gmail.com] Sent: Sunday, February 03, 2013 7:42 AM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? With solrclound all cores are collections. The collections API it just a wrapper to call the core api a million times with one command. to /solr/admin/cores?action=CREATEname=core1collection=core1shard=1 Basically your creating the shard again, after leader props have gone out. Solr will check ZK and find a core meeting that description, then simply get a copy of the index from the leader of that shard. On Feb 3, 2013, at 10:37 AM, Brett Hoerner br...@bretthoerner.com wrote: What is the inverse I'd use to re-create/load a core on another machine but make sure it's also known to SolrCloud/as a shard? On Sat, Feb 2, 2013 at 4:01 PM, Joseph Dale joey.d...@gmail.com wrote: To be more clear lets say bob it the leader of core 1. On bob do a /admin/cores?action=unloadname=core1. This removes the core/shard from bob, giving the other servers a chance to grab leader props. -Joey On Feb 2, 2013, at 11:27 AM, Brett Hoerner br...@bretthoerner.com wrote: Hi, I have a 5 server cluster running 1 collection with 20 shards, replication factor of 2. Earlier this week I had to do a rolling restart across the cluster, this worked great and the cluster stayed up the whole time. The problem is that the last node I restarted is now the leader of 0 shards, and is just holding replicas. I've noticed this node has abnormally high load average, while the other nodes (who have the same number of shards, but more leaders on average) are fine. First, I'm wondering if that loud could be related to being a 5x replica and 0x leader? Second, I was wondering if I could somehow flag single shards to re-elect a leader (or force a leader) so that I could more evenly distribute how many leader shards each physical server has running? Thanks. -- - Mark
RE: Is it possible to manually select a shard leader in a running SolrCloud?
Hey all, I feel having to unload the leader core to force an election is hacky, and as far as I know would still leave which node becomes the Leader to chance, ie I cannot guarantee NodeX becomes Leader 100% in all cases. Also, this imposes additional load temporarily. Is there a way to force the winner of the Election, and if not, is there a known feature-request for this? Cheers, Tim Vaillancourt -Original Message- From: Joseph Dale [mailto:joey.d...@gmail.com] Sent: Sunday, February 03, 2013 7:42 AM To: solr-user@lucene.apache.org Subject: Re: Is it possible to manually select a shard leader in a running SolrCloud? With solrclound all cores are collections. The collections API it just a wrapper to call the core api a million times with one command. to /solr/admin/cores?action=CREATEname=core1collection=core1shard=1 Basically your creating the shard again, after leader props have gone out. Solr will check ZK and find a core meeting that description, then simply get a copy of the index from the leader of that shard. On Feb 3, 2013, at 10:37 AM, Brett Hoerner br...@bretthoerner.com wrote: What is the inverse I'd use to re-create/load a core on another machine but make sure it's also known to SolrCloud/as a shard? On Sat, Feb 2, 2013 at 4:01 PM, Joseph Dale joey.d...@gmail.com wrote: To be more clear lets say bob it the leader of core 1. On bob do a /admin/cores?action=unloadname=core1. This removes the core/shard from bob, giving the other servers a chance to grab leader props. -Joey On Feb 2, 2013, at 11:27 AM, Brett Hoerner br...@bretthoerner.com wrote: Hi, I have a 5 server cluster running 1 collection with 20 shards, replication factor of 2. Earlier this week I had to do a rolling restart across the cluster, this worked great and the cluster stayed up the whole time. The problem is that the last node I restarted is now the leader of 0 shards, and is just holding replicas. I've noticed this node has abnormally high load average, while the other nodes (who have the same number of shards, but more leaders on average) are fine. First, I'm wondering if that loud could be related to being a 5x replica and 0x leader? Second, I was wondering if I could somehow flag single shards to re-elect a leader (or force a leader) so that I could more evenly distribute how many leader shards each physical server has running? Thanks.
Re: Is it possible to manually select a shard leader in a running SolrCloud?
What is the inverse I'd use to re-create/load a core on another machine but make sure it's also known to SolrCloud/as a shard? On Sat, Feb 2, 2013 at 4:01 PM, Joseph Dale joey.d...@gmail.com wrote: To be more clear lets say bob it the leader of core 1. On bob do a /admin/cores?action=unloadname=core1. This removes the core/shard from bob, giving the other servers a chance to grab leader props. -Joey On Feb 2, 2013, at 11:27 AM, Brett Hoerner br...@bretthoerner.com wrote: Hi, I have a 5 server cluster running 1 collection with 20 shards, replication factor of 2. Earlier this week I had to do a rolling restart across the cluster, this worked great and the cluster stayed up the whole time. The problem is that the last node I restarted is now the leader of 0 shards, and is just holding replicas. I've noticed this node has abnormally high load average, while the other nodes (who have the same number of shards, but more leaders on average) are fine. First, I'm wondering if that loud could be related to being a 5x replica and 0x leader? Second, I was wondering if I could somehow flag single shards to re-elect a leader (or force a leader) so that I could more evenly distribute how many leader shards each physical server has running? Thanks.
Re: Is it possible to manually select a shard leader in a running SolrCloud?
With solrclound all cores are collections. The collections API it just a wrapper to call the core api a million times with one command. to /solr/admin/cores?action=CREATEname=core1collection=core1shard=1 Basically your creating the shard again, after leader props have gone out. Solr will check ZK and find a core meeting that description, then simply get a copy of the index from the leader of that shard. On Feb 3, 2013, at 10:37 AM, Brett Hoerner br...@bretthoerner.com wrote: What is the inverse I'd use to re-create/load a core on another machine but make sure it's also known to SolrCloud/as a shard? On Sat, Feb 2, 2013 at 4:01 PM, Joseph Dale joey.d...@gmail.com wrote: To be more clear lets say bob it the leader of core 1. On bob do a /admin/cores?action=unloadname=core1. This removes the core/shard from bob, giving the other servers a chance to grab leader props. -Joey On Feb 2, 2013, at 11:27 AM, Brett Hoerner br...@bretthoerner.com wrote: Hi, I have a 5 server cluster running 1 collection with 20 shards, replication factor of 2. Earlier this week I had to do a rolling restart across the cluster, this worked great and the cluster stayed up the whole time. The problem is that the last node I restarted is now the leader of 0 shards, and is just holding replicas. I've noticed this node has abnormally high load average, while the other nodes (who have the same number of shards, but more leaders on average) are fine. First, I'm wondering if that loud could be related to being a 5x replica and 0x leader? Second, I was wondering if I could somehow flag single shards to re-elect a leader (or force a leader) so that I could more evenly distribute how many leader shards each physical server has running? Thanks.
Re: Is it possible to manually select a shard leader in a running SolrCloud?
As an update, it looks like the heavy load is in part because the node never catches back up with the other nodes. In SolrCloud UI it was yellow for a long time, then eventually grey, then back to yellow and orange. It never recovers as green. I should note this collection is very busy, indexing 5k+ small documents per second, but the nodes were all fine until I had to restart them and they had to re-sync. Here is the log since reboot: https://gist.github.com/396af4b217ce8f536db6 Any ideas? On Sat, Feb 2, 2013 at 10:27 AM, Brett Hoerner br...@bretthoerner.comwrote: Hi, I have a 5 server cluster running 1 collection with 20 shards, replication factor of 2. Earlier this week I had to do a rolling restart across the cluster, this worked great and the cluster stayed up the whole time. The problem is that the last node I restarted is now the leader of 0 shards, and is just holding replicas. I've noticed this node has abnormally high load average, while the other nodes (who have the same number of shards, but more leaders on average) are fine. First, I'm wondering if that loud could be related to being a 5x replica and 0x leader? Second, I was wondering if I could somehow flag single shards to re-elect a leader (or force a leader) so that I could more evenly distribute how many leader shards each physical server has running? Thanks.
Re: Is it possible to manually select a shard leader in a running SolrCloud?
Unload the cores you want on the leader. (Using the cores api, not the collections api). That will force a leader election, afterwards simply readd them. -Joey On Feb 2, 2013, at 11:27 AM, Brett Hoerner br...@bretthoerner.com wrote: Hi, I have a 5 server cluster running 1 collection with 20 shards, replication factor of 2. Earlier this week I had to do a rolling restart across the cluster, this worked great and the cluster stayed up the whole time. The problem is that the last node I restarted is now the leader of 0 shards, and is just holding replicas. I've noticed this node has abnormally high load average, while the other nodes (who have the same number of shards, but more leaders on average) are fine. First, I'm wondering if that loud could be related to being a 5x replica and 0x leader? Second, I was wondering if I could somehow flag single shards to re-elect a leader (or force a leader) so that I could more evenly distribute how many leader shards each physical server has running? Thanks.
Re: Is it possible to manually select a shard leader in a running SolrCloud?
To be more clear lets say bob it the leader of core 1. On bob do a /admin/cores?action=unloadname=core1. This removes the core/shard from bob, giving the other servers a chance to grab leader props. -Joey On Feb 2, 2013, at 11:27 AM, Brett Hoerner br...@bretthoerner.com wrote: Hi, I have a 5 server cluster running 1 collection with 20 shards, replication factor of 2. Earlier this week I had to do a rolling restart across the cluster, this worked great and the cluster stayed up the whole time. The problem is that the last node I restarted is now the leader of 0 shards, and is just holding replicas. I've noticed this node has abnormally high load average, while the other nodes (who have the same number of shards, but more leaders on average) are fine. First, I'm wondering if that loud could be related to being a 5x replica and 0x leader? Second, I was wondering if I could somehow flag single shards to re-elect a leader (or force a leader) so that I could more evenly distribute how many leader shards each physical server has running? Thanks.
