Using Autoscaling Simulation Framework to simulate a lost node in a cluster
Hello folks, has anyone tried to use the autoscaling simulation framework to simulate a lost node in a solr cluster? I was trying to do the following: 1.- Take a current production cluster state snapshout using bin/solr autoscaling -save 2.- Modify the clusterstate and livenodes json files in the generated folder to delete one of the nodes and its related replicas. 3.- Modify the clusterstate and livenodes json files in the generated folder to create a new empty node in the cluster. 4.- Run simulations playing with different policies and trigger waitFor values when adding a new node in the cluster replacing the lost one to see if the rules and triggers behave as expected. But I was wondering, is this something supported by the framework?. Or if there's a better approach to simulate this? Thanks in advance for any guidance/tips.
Re: New replica is being added to existing collections upon server reboot
Did you replace any node recently in the cluster? That downed replica may be related to that "gone/replaced" node and solr autoscaling added the new replicas on the new replacing node. It shows that downed replica if the replaced node hasn't been cleaned-up with the DELETENODE api command From: yaswanth kumar Sent: Thursday, August 27, 2020 12:14 PM To: solr-user@lucene.apache.org Subject: Re: New replica is being added to existing collections upon server reboot sorry there is a typo on earlier message.. I didn't make any changes in autoscalling policies/triggers On Thu, Aug 27, 2020 at 12:12 PM yaswanth kumar wrote: > Thanks for looking into this. > > I did used replication factor as 3 with autoAddReplica=true, and did make > any changes in autoscalling policies/triggers everything is defaults that > comes with solr. > > On Thu, Aug 27, 2020 at 11:50 AM Howard Gonzalez < > howard.gonza...@careerbuilder.com> wrote: > >> Hi, could you share the replication factor that you're using for those >> collections (in case they are NRT replicas)? DId you make any changes in >> autoscaling policies/triggers? >> >> From: yaswanth kumar >> Sent: Thursday, August 27, 2020 11:37 AM >> To: solr-user@lucene.apache.org >> Subject: New replica is being added to existing collections upon server >> reboot >> >> Can someone help me understand on why the below is happening? >> >> Solr: 8.2; Zookeer:3.5 >> >> One zookeeper + 3 solr nodes >> >> Initially created multiple collections with 3 replicas , indexed data >> everything looked great. >> >> We now restarted all 3 solr nodes, and we started the zookeeper and solr >> services , cloud came back good but on each collection we are now seeing a >> 4th replica and that is in down status. Other than this additional node >> every other operation is working fine. >> >> Please let me know on what case this could happen, also want to know if >> there is any easy way to get rid of the 4th newly added down solr replica >> for all collections. >> >> -- >> Thanks & Regards, >> Yaswanth Kumar Konathala. >> yaswanth...@gmail.com >> > > > -- > Thanks & Regards, > Yaswanth Kumar Konathala. > yaswanth...@gmail.com > -- Thanks & Regards, Yaswanth Kumar Konathala. yaswanth...@gmail.com
Re: New replica is being added to existing collections upon server reboot
Hi, could you share the replication factor that you're using for those collections (in case they are NRT replicas)? DId you make any changes in autoscaling policies/triggers? From: yaswanth kumar Sent: Thursday, August 27, 2020 11:37 AM To: solr-user@lucene.apache.org Subject: New replica is being added to existing collections upon server reboot Can someone help me understand on why the below is happening? Solr: 8.2; Zookeer:3.5 One zookeeper + 3 solr nodes Initially created multiple collections with 3 replicas , indexed data everything looked great. We now restarted all 3 solr nodes, and we started the zookeeper and solr services , cloud came back good but on each collection we are now seeing a 4th replica and that is in down status. Other than this additional node every other operation is working fine. Please let me know on what case this could happen, also want to know if there is any easy way to get rid of the 4th newly added down solr replica for all collections. -- Thanks & Regards, Yaswanth Kumar Konathala. yaswanth...@gmail.com
Re: How to Write Autoscaling Policy changes to Zookeeper/SolrCloud using the autoscaling Java API
Good morning! To add more context on the question, I can successfully use the Java API to build the list of new Clauses. However, the problem that I have is that I don't know how to "write" those changes back to solr using the Java API. I see there's a writeMap method in the Policy class however I can't find how to use it. Thanks in advance ________ From: Howard Gonzalez Sent: Friday, August 21, 2020 12:45 PM To: solr-user@lucene.apache.org Subject: How to Write Autoscaling Policy changes to Zookeeper/SolrCloud using the autoscaling Java API Hello. I am trying to use the autoscaling Java API to write some cluster policy changes to a Zookeeper/SolrCloud cluster. However, I can't find the right way to do it. I can get all the autoscaling cluster policy clauses using: autoScalingConfig.getPolicy.getClusterPolicy However, after getting all the right List of clauses, I don't know how to write those changes to the Zookeeper/Solr cluster using the Java API. Any guidance please? I know I can use the HTTP solr client to send a json request, but just wondering how to do it using the provided Java API. Thanks in advance
How to Write Autoscaling Policy changes to Zookeeper/SolrCloud using the autoscaling Java API
Hello. I am trying to use the autoscaling Java API to write some cluster policy changes to a Zookeeper/SolrCloud cluster. However, I can't find the right way to do it. I can get all the autoscaling cluster policy clauses using: autoScalingConfig.getPolicy.getClusterPolicy However, after getting all the right List of clauses, I don't know how to write those changes to the Zookeeper/Solr cluster using the Java API. Any guidance please? I know I can use the HTTP solr client to send a json request, but just wondering how to do it using the provided Java API. Thanks in advance
