[jira] [Commented] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521254#comment-16521254 ] Nitin Sharma commented on SOLR-9241: [~Fatalityap] I am currently not actively working on this patch. If I find time, i can look at the autoscale api and see how to rewrite some of my patches to fit that. > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma >Priority: Major > Labels: Cluster, SolrCloud > Fix For: 6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-4.6.patch, SOLR-9241-6.1.patch > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin...@gmail.com -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426189#comment-16426189 ] Nitin Sharma commented on SOLR-9241: Sorry i haven't kept this patch up with the latest solr releases. Last was 6.1. [~noble.paul] There is a basic version of resharding in this patch but it has not been ported yet. It does reduce/increase number of shards based on the following commands. Auto Shard - Dynamically shard a collection to any size. Smart Merge - Distributed Mode - Helps merging data from a larger shard setup into smaller one. (the source should be divisible by destination) > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma >Priority: Major > Labels: Cluster, SolrCloud > Fix For: 6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-4.6.patch, SOLR-9241-6.1.patch > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin...@gmail.com -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9241: --- Description: This is the v1 of the patch for Solrcloud Rebalance api (as described in http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to provide a zero downtime mechanism to perform data manipulation and efficient core allocation in solrcloud. This API was envisioned to be the base layer that enables Solrcloud to be an auto scaling platform. (and work in unison with other complementing monitoring and scaling features). Patch Status: === The patch is work in progress and incremental. We have done a few rounds of code clean up. We wanted to get the patch going first to get initial feed back. We will continue to work on making it more open source friendly and easily testable. Deployment Status: The platform is deployed in production at bloomreach and has been battle tested for large scale load. (millions of documents and hundreds of collections). Internals: = The internals of the API and performance : http://engineering.bloomreach.com/solrcloud-rebalance-api/ It is built on top of the admin collections API as an action (with various flavors). At a high level, the rebalance api provides 2 constructs: Scaling Strategy: Decides how to move the data. Every flavor has multiple options which can be reviewed in the api spec. Re-distribute - Move around data in the cluster based on capacity/allocation. Auto Shard - Dynamically shard a collection to any size. Smart Merge - Distributed Mode - Helps merging data from a larger shard setup into smaller one. (the source should be divisible by destination) Scale up - Add replicas on the fly Scale Down - Remove replicas on the fly Allocation Strategy: Decides where to put the data. (Nodes with least cores, Nodes that do not have this collection etc). Custom implementations can be built on top as well. One other example is Availability Zone aware. Distribute data such that every replica is placed on different availability zone to support HA. Detailed API Spec: https://github.com/bloomreach/solrcloud-rebalance-api Contributors: = Nitin Sharma Suruchi Shah Questions/Comments: = You can reach me at nitin...@gmail.com was: This is the v1 of the patch for Solrcloud Rebalance api (as described in http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to provide a zero downtime mechanism to perform data manipulation and efficient core allocation in solrcloud. This API was envisioned to be the base layer that enables Solrcloud to be an auto scaling platform. (and work in unison with other complementing monitoring and scaling features). Patch Status: === The patch is work in progress and incremental. We have done a few rounds of code clean up. We wanted to get the patch going first to get initial feed back. We will continue to work on making it more open source friendly and easily testable. Deployment Status: The platform is deployed in production at bloomreach and has been battle tested for large scale load. (millions of documents and hundreds of collections). Internals: = The internals of the API and performance : http://engineering.bloomreach.com/solrcloud-rebalance-api/ It is built on top of the admin collections API as an action (with various flavors). At a high level, the rebalance api provides 2 constructs: Scaling Strategy: Decides how to move the data. Every flavor has multiple options which can be reviewed in the api spec. Re-distribute - Move around data in the cluster based on capacity/allocation. Auto Shard - Dynamically shard a collection to any size. Smart Merge - Distributed Mode - Helps merging data from a larger shard setup into smaller one. (the source should be divisible by destination) Scale up - Add replicas on the fly Scale Down - Remove replicas on the fly Allocation Strategy: Decides where to put the data. (Nodes with least cores, Nodes that do not have this collection etc). Custom implementations can be built on top as well. One other example is Availability Zone aware. Distribute data such that every replica is placed on different availability zone to support HA. Detailed API Spec: https://github.com/bloomreach/solrcloud-rebalance-api Contributors: = Nitin Sharma Suruchi Shah Questions/Comments: = You can reach me at nitin.sha...@bloomreach.com > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL:
[jira] [Commented] (SOLR-9319) DELETEREPLICA should be able to accept just count and remove replicas intelligenty
[ https://issues.apache.org/jira/browse/SOLR-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453783#comment-15453783 ] Nitin Sharma commented on SOLR-9319: Thanks! > DELETEREPLICA should be able to accept just count and remove replicas > intelligenty > --- > > Key: SOLR-9319 > URL: https://issues.apache.org/jira/browse/SOLR-9319 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DeleteReplicaPatch.jpg, Delete_Replica_count_1.jpg, > Delete_Replica_invalid.jpg, Delete_Replica_with_only_1replica.jpg, > Delete_replica_count2.jpg, Delte_replica_after.jpg, Delte_replica_before.jpg, > Old_deletereplica_api_works.jpg, SOLR-9310.patch, SOLR-9319.patch, > SOLR-9319.patch, SOLR-9319.patch, SOLR-9319.patch, Screen Shot 2016-08-26 at > 12.59.16 PM.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9319) DELETEREPLICA should be able to accept just count and remove replicas intelligenty
[ https://issues.apache.org/jira/browse/SOLR-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450410#comment-15450410 ] Nitin Sharma commented on SOLR-9319: [~noble.paul] Any updates to the patch needed? > DELETEREPLICA should be able to accept just count and remove replicas > intelligenty > --- > > Key: SOLR-9319 > URL: https://issues.apache.org/jira/browse/SOLR-9319 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DeleteReplicaPatch.jpg, Delete_Replica_count_1.jpg, > Delete_Replica_invalid.jpg, Delete_Replica_with_only_1replica.jpg, > Delete_replica_count2.jpg, Delte_replica_after.jpg, Delte_replica_before.jpg, > Old_deletereplica_api_works.jpg, SOLR-9310.patch, SOLR-9319.patch, > SOLR-9319.patch, SOLR-9319.patch, SOLR-9319.patch, Screen Shot 2016-08-26 at > 12.59.16 PM.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9319) DELETEREPLICA should be able to accept just count and remove replicas intelligenty
[ https://issues.apache.org/jira/browse/SOLR-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9319: --- Attachment: SOLR-9319.patch Updated patch > DELETEREPLICA should be able to accept just count and remove replicas > intelligenty > --- > > Key: SOLR-9319 > URL: https://issues.apache.org/jira/browse/SOLR-9319 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DeleteReplicaPatch.jpg, Delete_Replica_count_1.jpg, > Delete_Replica_invalid.jpg, Delete_Replica_with_only_1replica.jpg, > Delete_replica_count2.jpg, Delte_replica_after.jpg, Delte_replica_before.jpg, > Old_deletereplica_api_works.jpg, SOLR-9310.patch, SOLR-9319.patch, > SOLR-9319.patch, SOLR-9319.patch, SOLR-9319.patch, Screen Shot 2016-08-26 at > 12.59.16 PM.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9319) DELETEREPLICA should be able to accept just count and remove replicas intelligenty
[ https://issues.apache.org/jira/browse/SOLR-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437799#comment-15437799 ] Nitin Sharma commented on SOLR-9319: [~noble.paul] Added one test in async mode and the other in sync mode. Closed the old pull request #70. Raised a new one #72. > DELETEREPLICA should be able to accept just count and remove replicas > intelligenty > --- > > Key: SOLR-9319 > URL: https://issues.apache.org/jira/browse/SOLR-9319 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DeleteReplicaPatch.jpg, Delete_Replica_count_1.jpg, > Delete_Replica_invalid.jpg, Delete_Replica_with_only_1replica.jpg, > Delete_replica_count2.jpg, Delte_replica_after.jpg, Delte_replica_before.jpg, > Old_deletereplica_api_works.jpg, SOLR-9310.patch, SOLR-9319.patch, > SOLR-9319.patch, SOLR-9319.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9319) DELETEREPLICA should be able to accept just count and remove replicas intelligenty
[ https://issues.apache.org/jira/browse/SOLR-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431763#comment-15431763 ] Nitin Sharma edited comment on SOLR-9319 at 8/22/16 11:05 PM: -- [~noble.paul] Rebased on master. Ran E2e Tests and unit tests. was (Author: nitin.sharma): Rebased on master. Ran E2e Tests and unit tests > DELETEREPLICA should be able to accept just count and remove replicas > intelligenty > --- > > Key: SOLR-9319 > URL: https://issues.apache.org/jira/browse/SOLR-9319 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DeleteReplicaPatch.jpg, Delete_Replica_count_1.jpg, > Delete_Replica_invalid.jpg, Delete_Replica_with_only_1replica.jpg, > Delete_replica_count2.jpg, Delte_replica_after.jpg, Delte_replica_before.jpg, > Old_deletereplica_api_works.jpg, SOLR-9310.patch, SOLR-9319.patch, > SOLR-9319.patch, SOLR-9319.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9319) DELETEREPLICA should be able to accept just count and remove replicas intelligenty
[ https://issues.apache.org/jira/browse/SOLR-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9319: --- Attachment: SOLR-9319.patch Rebased on master. Ran E2e Tests and unit tests > DELETEREPLICA should be able to accept just count and remove replicas > intelligenty > --- > > Key: SOLR-9319 > URL: https://issues.apache.org/jira/browse/SOLR-9319 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DeleteReplicaPatch.jpg, Delete_Replica_count_1.jpg, > Delete_Replica_invalid.jpg, Delete_Replica_with_only_1replica.jpg, > Delete_replica_count2.jpg, Delte_replica_after.jpg, Delte_replica_before.jpg, > Old_deletereplica_api_works.jpg, SOLR-9310.patch, SOLR-9319.patch, > SOLR-9319.patch, SOLR-9319.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Issue Comment Deleted] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9241: --- Comment: was deleted (was: Rebased on top of master. Ran tests both unit and E2e) > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-4.6.patch, SOLR-9241-6.1.patch > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431757#comment-15431757 ] Nitin Sharma edited comment on SOLR-9241 at 8/22/16 11:03 PM: -- Rebased on top of master. Ran tests both unit and E2e was (Author: nitin.sharma): Rebased on top of master > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-4.6.patch, SOLR-9241-6.1.patch > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9241: --- Attachment: SOLR-9319.patch Rebased on top of master > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-4.6.patch, SOLR-9241-6.1.patch > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9241: --- Attachment: (was: SOLR-9319.patch) > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-4.6.patch, SOLR-9241-6.1.patch > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9319) DELETEREPLICA should be able to accept just count and remove replicas intelligenty
[ https://issues.apache.org/jira/browse/SOLR-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431249#comment-15431249 ] Nitin Sharma commented on SOLR-9319: Ok i will rebase on master again. BTW i think you uploaded the patch for 9310 here. > DELETEREPLICA should be able to accept just count and remove replicas > intelligenty > --- > > Key: SOLR-9319 > URL: https://issues.apache.org/jira/browse/SOLR-9319 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DeleteReplicaPatch.jpg, Delete_Replica_count_1.jpg, > Delete_Replica_invalid.jpg, Delete_Replica_with_only_1replica.jpg, > Delete_replica_count2.jpg, Delte_replica_after.jpg, Delte_replica_before.jpg, > Old_deletereplica_api_works.jpg, SOLR-9310.patch, SOLR-9319.patch, > SOLR-9319.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9319) DELETEREPLICA should be able to accept just count and remove replicas intelligenty
[ https://issues.apache.org/jira/browse/SOLR-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428268#comment-15428268 ] Nitin Sharma commented on SOLR-9319: It is already rebased on top of current master. Did you mean commit to trunk? > DELETEREPLICA should be able to accept just count and remove replicas > intelligenty > --- > > Key: SOLR-9319 > URL: https://issues.apache.org/jira/browse/SOLR-9319 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DeleteReplicaPatch.jpg, Delete_Replica_count_1.jpg, > Delete_Replica_invalid.jpg, Delete_Replica_with_only_1replica.jpg, > Delete_replica_count2.jpg, Delte_replica_after.jpg, Delte_replica_before.jpg, > Old_deletereplica_api_works.jpg, SOLR-9319.patch, SOLR-9319.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9319) DELETEREPLICA should be able to accept just count and remove replicas intelligenty
[ https://issues.apache.org/jira/browse/SOLR-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425887#comment-15425887 ] Nitin Sharma edited comment on SOLR-9319 at 8/18/16 4:24 AM: - With Tests. Ran ant test with full suite and the ant test with -Dtestcase=DeleteReplicaTest to verify all passed. was (Author: nitin.sharma): With Tests > DELETEREPLICA should be able to accept just count and remove replicas > intelligenty > --- > > Key: SOLR-9319 > URL: https://issues.apache.org/jira/browse/SOLR-9319 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DeleteReplicaPatch.jpg, Delete_Replica_count_1.jpg, > Delete_Replica_invalid.jpg, Delete_Replica_with_only_1replica.jpg, > Delete_replica_count2.jpg, Delte_replica_after.jpg, Delte_replica_before.jpg, > Old_deletereplica_api_works.jpg, SOLR-9319.patch, SOLR-9319.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9319) DELETEREPLICA should be able to accept just count and remove replicas intelligenty
[ https://issues.apache.org/jira/browse/SOLR-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9319: --- Attachment: SOLR-9319.patch With Tests > DELETEREPLICA should be able to accept just count and remove replicas > intelligenty > --- > > Key: SOLR-9319 > URL: https://issues.apache.org/jira/browse/SOLR-9319 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DeleteReplicaPatch.jpg, Delete_Replica_count_1.jpg, > Delete_Replica_invalid.jpg, Delete_Replica_with_only_1replica.jpg, > Delete_replica_count2.jpg, Delte_replica_after.jpg, Delte_replica_before.jpg, > Old_deletereplica_api_works.jpg, SOLR-9319.patch, SOLR-9319.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9319) DELETEREPLICA should be able to accept just count and remove replicas intelligenty
[ https://issues.apache.org/jira/browse/SOLR-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425451#comment-15425451 ] Nitin Sharma commented on SOLR-9319: Sure. Let me try that out. > DELETEREPLICA should be able to accept just count and remove replicas > intelligenty > --- > > Key: SOLR-9319 > URL: https://issues.apache.org/jira/browse/SOLR-9319 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DeleteReplicaPatch.jpg, Delete_Replica_count_1.jpg, > Delete_Replica_invalid.jpg, Delete_Replica_with_only_1replica.jpg, > Delete_replica_count2.jpg, Delte_replica_after.jpg, Delte_replica_before.jpg, > Old_deletereplica_api_works.jpg, SOLR-9319.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9319) DELETEREPLICA should be able to accept just count and remove replicas intelligenty
[ https://issues.apache.org/jira/browse/SOLR-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423852#comment-15423852 ] Nitin Sharma commented on SOLR-9319: [~noble.paul] Please let me know if this is missing anything > DELETEREPLICA should be able to accept just count and remove replicas > intelligenty > --- > > Key: SOLR-9319 > URL: https://issues.apache.org/jira/browse/SOLR-9319 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DeleteReplicaPatch.jpg, Delete_Replica_count_1.jpg, > Delete_Replica_invalid.jpg, Delete_Replica_with_only_1replica.jpg, > Delete_replica_count2.jpg, Delte_replica_after.jpg, Delte_replica_before.jpg, > Old_deletereplica_api_works.jpg, SOLR-9319.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9319) DELETEREPLICA should be able to accept just count and remove replicas intelligenty
[ https://issues.apache.org/jira/browse/SOLR-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9319: --- Attachment: DeleteReplicaPatch.jpg Delete_Replica_count_1.jpg Delete_Replica_invalid.jpg Delete_Replica_with_only_1replica.jpg Delte_replica_before.jpg Delete_replica_count2.jpg Delte_replica_after.jpg Old_deletereplica_api_works.jpg Screenshots with a) API with Count param and intelligent removal b) API with count and only 1 replica remaining c) API with count and more replicas are requested than present d) Old API call without count param working as expected > DELETEREPLICA should be able to accept just count and remove replicas > intelligenty > --- > > Key: SOLR-9319 > URL: https://issues.apache.org/jira/browse/SOLR-9319 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DeleteReplicaPatch.jpg, Delete_Replica_count_1.jpg, > Delete_Replica_invalid.jpg, Delete_Replica_with_only_1replica.jpg, > Delete_replica_count2.jpg, Delte_replica_after.jpg, Delte_replica_before.jpg, > Old_deletereplica_api_works.jpg, SOLR-9319.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9319) DELETEREPLICA should be able to accept just count and remove replicas intelligenty
[ https://issues.apache.org/jira/browse/SOLR-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9319: --- Attachment: SOLR-9319.patch > DELETEREPLICA should be able to accept just count and remove replicas > intelligenty > --- > > Key: SOLR-9319 > URL: https://issues.apache.org/jira/browse/SOLR-9319 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: SOLR-9319.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9319) DELETEREPLICA should be able to accept just count and remove replicas intelligenty
[ https://issues.apache.org/jira/browse/SOLR-9319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418320#comment-15418320 ] Nitin Sharma commented on SOLR-9319: [~noble.paul] When you mean intelligently, I am assuming the following 1) If it is the only replica it would not get deleted. 2) If there are multiple replicas, then it will remove the replica that is not the leader (to lessen operations in the cluster) Let me know if there is anything else that is needed. I am testing out patch for this as we speak. > DELETEREPLICA should be able to accept just count and remove replicas > intelligenty > --- > > Key: SOLR-9319 > URL: https://issues.apache.org/jira/browse/SOLR-9319 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9320) A REPLACENODE command to decommission an existing node with another new node
[ https://issues.apache.org/jira/browse/SOLR-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412691#comment-15412691 ] Nitin Sharma commented on SOLR-9320: Most of the patch looks good. In ReplaceNodeCmd, instead of calling DeleteNodeCmd.deletereplicas, can you just called DELETENODE cmd on the source node itself? Looks much cleaner that way and one command doesnt need to know the details of the other cmd. > A REPLACENODE command to decommission an existing node with another new node > > > Key: SOLR-9320 > URL: https://issues.apache.org/jira/browse/SOLR-9320 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Attachments: DELETENODE.jpeg, REPLACENODE_After.jpeg, > REPLACENODE_Before.jpeg, REPLACENODE_call_response.jpeg, SOLR-9320.patch, > SOLR-9320.patch, SOLR-9320.patch, SOLR-9320.patch, SOLR-9320.patch, > SOLR-9320.patch, SOLR-9320.patch > > > The command should accept a source node and target node. recreate the > replicas in source node in the target and do a DLETENODE of source node -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9320) A REPLACENODE command to decommission an existing node with another new node
[ https://issues.apache.org/jira/browse/SOLR-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410678#comment-15410678 ] Nitin Sharma commented on SOLR-9320: Functionality seems to be fine to me. The spec you mentioned in the description was that REPLACE does a create and then invokes a DELETENODE. The recent patch seems to do iterate and do a delete replica instead of having a DELETENODE action and calling that. Hence the original question. I tried the first one and it works fine. I will try your new patch. > A REPLACENODE command to decommission an existing node with another new node > > > Key: SOLR-9320 > URL: https://issues.apache.org/jira/browse/SOLR-9320 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Attachments: DELETENODE.jpeg, REPLACENODE_After.jpeg, > REPLACENODE_Before.jpeg, REPLACENODE_call_response.jpeg, SOLR-9320.patch, > SOLR-9320.patch, SOLR-9320.patch, SOLR-9320.patch > > > The command should accept a source node and target node. recreate the > replicas in source node in the target and do a DLETENODE of source node -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9320) A REPLACENODE command to decommission an existing node with another new node
[ https://issues.apache.org/jira/browse/SOLR-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15410472#comment-15410472 ] Nitin Sharma commented on SOLR-9320: The patch i attached follows the original spec you proposed. Replace node after creating new replicas will call "DELETENODE" on the destination host which will remove all cores on the node. Can you merge this with that as well? Thanks. I will patch this on top of 6.1 and give it a shot > A REPLACENODE command to decommission an existing node with another new node > > > Key: SOLR-9320 > URL: https://issues.apache.org/jira/browse/SOLR-9320 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Attachments: DELETENODE.jpeg, REPLACENODE_After.jpeg, > REPLACENODE_Before.jpeg, REPLACENODE_call_response.jpeg, SOLR-9320.patch, > SOLR-9320.patch, SOLR-9320.patch > > > The command should accept a source node and target node. recreate the > replicas in source node in the target and do a DLETENODE of source node -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9320) A REPLACENODE command to decommission an existing node with another new node
[ https://issues.apache.org/jira/browse/SOLR-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404456#comment-15404456 ] Nitin Sharma commented on SOLR-9320: Would be good to establish context on when this command should/can be run. In our current version of rebalance, we use this to switch serving nodes in production and hence the time taken to migrate a node is limited (time bound). If this is considered an offline process, then may be serial works just fine. I agree with Erick on throttling but we can do that at a system level (unix traffic shaping tc etc). That will leave the bandwidth management outside the responsibility of solr. The same node serving solr can be used to run other services in some production setups. Keeping that outside solr will put that onus on the maintainer. Just my $0.02 > A REPLACENODE command to decommission an existing node with another new node > > > Key: SOLR-9320 > URL: https://issues.apache.org/jira/browse/SOLR-9320 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DELETENODE.jpeg, REPLACENODE_After.jpeg, > REPLACENODE_Before.jpeg, REPLACENODE_call_response.jpeg, SOLR-9320.patch, > SOLR-9320.patch > > > The command should accept a source node and target node. recreate the > replicas in source node in the target and do a DLETENODE of source node -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9320) A REPLACENODE command to decommission an existing node with another new node
[ https://issues.apache.org/jira/browse/SOLR-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403326#comment-15403326 ] Nitin Sharma edited comment on SOLR-9320 at 8/2/16 4:30 AM: Thanks for clarifying on async. Isnt that only for executing multiple api in parallel. If you want parallelism within the api, you still need your own exec service right? Reg 500 collections: When you run replace node on a node x and the node happens to have 500 cores, then all these cores need to be moved to the destination (and then removed from source). If you have too many cores, adding a new replica for every core on the destination host will take time. Wondering if you ran any tests with such high capacity was (Author: nitin.sharma): Thanks for clarifying on async. Reg 500 collections: When you run replace node on a node x and the node happens to have 500 cores, then all these cores need to be moved to the destination (and then removed from source). If you have too many cores, adding a new replica for every core on the destination host will take time. Wondering if you ran any tests with such high capacity > A REPLACENODE command to decommission an existing node with another new node > > > Key: SOLR-9320 > URL: https://issues.apache.org/jira/browse/SOLR-9320 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DELETENODE.jpeg, REPLACENODE_After.jpeg, > REPLACENODE_Before.jpeg, REPLACENODE_call_response.jpeg, SOLR-9320.patch, > SOLR-9320.patch > > > The command should accept a source node and target node. recreate the > replicas in source node in the target and do a DLETENODE of source node -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9320) A REPLACENODE command to decommission an existing node with another new node
[ https://issues.apache.org/jira/browse/SOLR-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403326#comment-15403326 ] Nitin Sharma commented on SOLR-9320: Thanks for clarifying on async. Reg 500 collections: When you run replace node on a node x and the node happens to have 500 cores, then all these cores need to be moved to the destination (and then removed from source). If you have too many cores, adding a new replica for every core on the destination host will take time. Wondering if you ran any tests with such high capacity > A REPLACENODE command to decommission an existing node with another new node > > > Key: SOLR-9320 > URL: https://issues.apache.org/jira/browse/SOLR-9320 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DELETENODE.jpeg, REPLACENODE_After.jpeg, > REPLACENODE_Before.jpeg, REPLACENODE_call_response.jpeg, SOLR-9320.patch, > SOLR-9320.patch > > > The command should accept a source node and target node. recreate the > replicas in source node in the target and do a DLETENODE of source node -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9320) A REPLACENODE command to decommission an existing node with another new node
[ https://issues.apache.org/jira/browse/SOLR-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402587#comment-15402587 ] Nitin Sharma commented on SOLR-9320: I have a concern when this comes to scaling to 100s of collections. If you have 100+ collections, this command will eventually timeout (> 3 mins) and this does this in serial. The patch i sent (has multi threading capability), I ran with 500 collections and it finished around 4+ mins. Can you share the numbers with collections of that scale? > A REPLACENODE command to decommission an existing node with another new node > > > Key: SOLR-9320 > URL: https://issues.apache.org/jira/browse/SOLR-9320 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DELETENODE.jpeg, REPLACENODE_After.jpeg, > REPLACENODE_Before.jpeg, REPLACENODE_call_response.jpeg, SOLR-9320.patch, > SOLR-9320.patch > > > The command should accept a source node and target node. recreate the > replicas in source node in the target and do a DLETENODE of source node -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9320) A REPLACENODE command to decommission an existing node with another new node
[ https://issues.apache.org/jira/browse/SOLR-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396174#comment-15396174 ] Nitin Sharma commented on SOLR-9320: This has the patch for both REPLACENODE and DELETENODE inside it. I can reduce the num files again and re-upload if you like. > A REPLACENODE command to decommission an existing node with another new node > > > Key: SOLR-9320 > URL: https://issues.apache.org/jira/browse/SOLR-9320 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DELETENODE.jpeg, REPLACENODE_After.jpeg, > REPLACENODE_Before.jpeg, REPLACENODE_call_response.jpeg, SOLR-9320.patch > > > The command should accept a source node and target node. recreate the > replicas in source node in the target and do a DLETENODE of source node -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9320) A REPLACENODE command to decommission an existing node with another new node
[ https://issues.apache.org/jira/browse/SOLR-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396152#comment-15396152 ] Nitin Sharma commented on SOLR-9320: [~noble.paul] Can you explain about the complex part? The code does what was mentioned in the spec. It identifies all the cores on the node to be replaced and recreates them on the destination node. After that it calls "DELETENODE" on the source node. Which part of this complex? The multithreaded part? > A REPLACENODE command to decommission an existing node with another new node > > > Key: SOLR-9320 > URL: https://issues.apache.org/jira/browse/SOLR-9320 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DELETENODE.jpeg, REPLACENODE_After.jpeg, > REPLACENODE_Before.jpeg, REPLACENODE_call_response.jpeg, SOLR-9320.patch > > > The command should accept a source node and target node. recreate the > replicas in source node in the target and do a DLETENODE of source node -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9318) A DELETENODE command that should delete all replicas in that node
[ https://issues.apache.org/jira/browse/SOLR-9318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390536#comment-15390536 ] Nitin Sharma commented on SOLR-9318: Refer to [SOLR-9320] for the patch for DELETENODE. > A DELETENODE command that should delete all replicas in that node > - > > Key: SOLR-9318 > URL: https://issues.apache.org/jira/browse/SOLR-9318 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > > The command should look in all collections , find out replicas hosted in that > node and remove them -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9320) A REPLACENODE command to decommission an existing node with another new node
[ https://issues.apache.org/jira/browse/SOLR-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9320: --- Attachment: REPLACENODE_Before.jpeg REPLACENODE_call_response.jpeg REPLACENODE_After.jpeg DELETENODE.jpeg SOLR-9320.patch Patch for REPLACENODE & DELETENODE api as per spec. DELETENODE- Deletes all cores on the given node. REPLACENODE - Replaces all cores from a source node to a dest node and then calls DELETENODE on the source node. Attached screenshots against test cluster with api calls & before/after status of the cluster > A REPLACENODE command to decommission an existing node with another new node > > > Key: SOLR-9320 > URL: https://issues.apache.org/jira/browse/SOLR-9320 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > Attachments: DELETENODE.jpeg, REPLACENODE_After.jpeg, > REPLACENODE_Before.jpeg, REPLACENODE_call_response.jpeg, SOLR-9320.patch > > > The command should accept a source node and target node. recreate the > replicas in source node in the target and do a DLETENODE of source node -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390486#comment-15390486 ] Nitin Sharma edited comment on SOLR-9241 at 7/23/16 3:03 AM: - I looked back some stats and found that I have run this upto close to 1T. 10 shards of 100 G each. We split that into 20 shards using merge based auto shard - Took around 1 hour but works reliably. Another index of size 2T with 50 shards (of 40 G each). We merged that into 10 shards using the smart merge strategy. That took around 10-15 mins. (Depending on machine type, ssd and network bandwidth) was (Author: nitin.sharma): I looked back some stats and found that I have run this upto close to 1T. 10 shards of 10 G each. We split that into 20 shards using merge based auto shard - Took around 1 hour but works reliably. Another index of size 2T with 50 shards (of 40 G each). We merged that into 10 shards using the smart merge strategy. That took around 10-15 mins. (Depending on machine type, ssd and network bandwidth) > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-4.6.patch, SOLR-9241-6.1.patch > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15390486#comment-15390486 ] Nitin Sharma commented on SOLR-9241: I looked back some stats and found that I have run this upto close to 1T. 10 shards of 10 G each. We split that into 20 shards using merge based auto shard - Took around 1 hour but works reliably. Another index of size 2T with 50 shards (of 40 G each). We merged that into 10 shards using the smart merge strategy. That took around 10-15 mins. (Depending on machine type, ssd and network bandwidth) > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-4.6.patch, SOLR-9241-6.1.patch > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387742#comment-15387742 ] Nitin Sharma commented on SOLR-9241: I have tested it with upto 200G index and split from 10-50 shards. It takes around 20 mins but finished everytime. We have another api called SMART_MERGE (used to go from larger shard setup to smaller shard), that is way more optimized in terms of computing num of merges required. It merges around 100G of index from 60 shards in 10 shards in around 7.5 mins. > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-4.6.patch, SOLR-9241-6.1.patch > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9317) ADDREPLICA command should be more flexible and add 'n' replicas to a collection,shard
[ https://issues.apache.org/jira/browse/SOLR-9317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387168#comment-15387168 ] Nitin Sharma commented on SOLR-9317: This is a bit vague. What does automatically identify mean? Will it be based on nodes with least cores or nodes that do not have a replica for a given collection. If we take it a step further it could also be based on nodes that have the least disk usage. What was in your mind? > ADDREPLICA command should be more flexible and add 'n' replicas to a > collection,shard > - > > Key: SOLR-9317 > URL: https://issues.apache.org/jira/browse/SOLR-9317 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > > It should automatically identify the nodes where these replicas should be > created as well -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9320) A REPLACENODE command to decommission an existing node with another new node
[ https://issues.apache.org/jira/browse/SOLR-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386191#comment-15386191 ] Nitin Sharma commented on SOLR-9320: Thanks for clarifying. I am aligning towards option a) as well (for simplicity and performance) but wanted to confirm the semantics. > A REPLACENODE command to decommission an existing node with another new node > > > Key: SOLR-9320 > URL: https://issues.apache.org/jira/browse/SOLR-9320 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > > The command should accept a source node and target node. recreate the > replicas in source node in the target and do a DLETENODE of source node -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9320) A REPLACENODE command to decommission an existing node with another new node
[ https://issues.apache.org/jira/browse/SOLR-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385323#comment-15385323 ] Nitin Sharma commented on SOLR-9320: Clarification on this. When you mean recreate, does the naming matter? Lets say x_shard1_replica1 is on source node, if you want to move it to destination node, we can a) either create a new replica (x_shard1_replica2) and delete the source. That will leave uneven naming conventions in the cluster. (there will not be a replica1 but a replica2). b) Preserve the exact same name as the replica in source node. We can achieve this by creating a temp replica on destination first, deleting the replica on source, recreating the replica (with same name) on destination and then cleaning up the temp. Option (b) can be thought of as a migrate core. Let me know which sounds more usable. > A REPLACENODE command to decommission an existing node with another new node > > > Key: SOLR-9320 > URL: https://issues.apache.org/jira/browse/SOLR-9320 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: Noble Paul > Fix For: 6.1 > > > The command should accept a source node and target node. recreate the > replicas in source node in the target and do a DLETENODE of source node -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15384337#comment-15384337 ] Nitin Sharma commented on SOLR-9241: [~noble.paul] Thanks for the feedback. Those are some major changes to be made. I will submit a patch only for REDISTRIBUTE/REPLACE with the recommended changes and we can iterate from there. A few clarifications 1) REPLACENODE: Do you want this to be a seperate admin action? or as a sub action of REBALANCE? Ex. /solr/admin?action=REPLACENODE or /solr/admin/collections?action=REBALANCE_strategy=REPLACENODE? 2) What is the default behavior of the replica replacement strategy? Does it pick unused nodes or just uses round robin to pick the next replica? We have unused and least used as 2 allocation strategies. I can fold them as separate replica replacement strategies if we like. > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-4.6.patch, SOLR-9241-6.1.patch > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9241: --- Affects Version/s: (was: 4.6.1) 6.1 > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-4.6.patch, SOLR-9241-6.1.patch > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9241: --- Affects Version/s: (was: 4.6.1) > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-4.6.patch, SOLR-9241-6.1.patch > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9241: --- Affects Version/s: 4.6.1 > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-4.6.patch, SOLR-9241-6.1.patch > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9241: --- Fix Version/s: (was: 4.6.1) 6.1 > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-4.6.patch, SOLR-9241-6.1.patch > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9241: --- Attachment: SOLR-9241-4.6.patch > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 4.6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 4.6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-4.6.patch, SOLR-9241-6.1.patch > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9241: --- Attachment: (was: SOLR-9241-4.6.1.patch) > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 4.6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 4.6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-6.1.patch > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9241: --- Attachment: (was: rebalance.diff) > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 4.6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 4.6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-6.1.patch > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9241: --- Attachment: SOLR-9241-6.1.patch SOLR-9241-4.6.1.patch Redistribute_Before.jpeg Redistribute_After.jpeg Redistribute_call.jpeg Replace_Call.jpeg Replace_Before.jpeg Replace_After.jpeg This contains two changes. 1) SOLR-9241-4.6.1 (patch) - The original diff has been modified to adhere to the naming conventions. 2) SOLR-9241-6.1 (patch) - The patch of Redistribute, Replace scaling strategies of the rebalance api on top of 6.1. Also attached are screenshots of the api calls and cluster state before vs after. > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 4.6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 4.6.1 > > Attachments: Redistribute_After.jpeg, Redistribute_Before.jpeg, > Redistribute_call.jpeg, Replace_After.jpeg, Replace_Before.jpeg, > Replace_Call.jpeg, SOLR-9241-4.6.1.patch, SOLR-9241-6.1.patch, rebalance.diff > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368480#comment-15368480 ] Nitin Sharma commented on SOLR-9241: [~noblepaul] Let me know if you would want to split this feature wise? (A separate patch for every scaling strategy ?) Kindly advise. > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 4.6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 4.6.1 > > Attachments: rebalance.diff > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15344786#comment-15344786 ] Nitin Sharma commented on SOLR-9241: I am in parallel trying to port my patch to master and testing out the API. I will update the jira either if I am successful or if I run into issues in porting. Thanks for the feedback! Joel: I will update the patch with the right naming convention too. > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 4.6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 4.6.1 > > Attachments: rebalance.diff > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api (as described in > http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at > Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to > provide a zero downtime mechanism to perform data manipulation and efficient > core allocation in solrcloud. This API was envisioned to be the base layer > that enables Solrcloud to be an auto scaling platform. (and work in unison > with other complementing monitoring and scaling features). > Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > Contributors: > = > Nitin Sharma > Suruchi Shah > Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9241: --- Description: This is the v1 of the patch for Solrcloud Rebalance api (as described in http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to provide a zero downtime mechanism to perform data manipulation and efficient core allocation in solrcloud. This API was envisioned to be the base layer that enables Solrcloud to be an auto scaling platform. (and work in unison with other complementing monitoring and scaling features). Patch Status: === The patch is work in progress and incremental. We have done a few rounds of code clean up. We wanted to get the patch going first to get initial feed back. We will continue to work on making it more open source friendly and easily testable. Deployment Status: The platform is deployed in production at bloomreach and has been battle tested for large scale load. (millions of documents and hundreds of collections). Internals: = The internals of the API and performance : http://engineering.bloomreach.com/solrcloud-rebalance-api/ It is built on top of the admin collections API as an action (with various flavors). At a high level, the rebalance api provides 2 constructs: Scaling Strategy: Decides how to move the data. Every flavor has multiple options which can be reviewed in the api spec. Re-distribute - Move around data in the cluster based on capacity/allocation. Auto Shard - Dynamically shard a collection to any size. Smart Merge - Distributed Mode - Helps merging data from a larger shard setup into smaller one. (the source should be divisible by destination) Scale up - Add replicas on the fly Scale Down - Remove replicas on the fly Allocation Strategy: Decides where to put the data. (Nodes with least cores, Nodes that do not have this collection etc). Custom implementations can be built on top as well. One other example is Availability Zone aware. Distribute data such that every replica is placed on different availability zone to support HA. # Detailed API Spec: https://github.com/bloomreach/solrcloud-rebalance-api # Contributors: = Nitin Sharma Suruchi Shah # Questions/Comments: = You can reach me at nitin.sha...@bloomreach.com was: This is the v1 of the patch for Solrcloud Rebalance api (as described in http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to provide a zero downtime mechanism to perform data manipulation and efficient core allocation in solrcloud. This API was envisioned to be the base layer that enables Solrcloud to be an auto scaling platform. (and work in unison with other complementing monitoring and scaling features). # Patch Status: === The patch is work in progress and incremental. We have done a few rounds of code clean up. We wanted to get the patch going first to get initial feed back. We will continue to work on making it more open source friendly and easily testable. # Deployment Status: The platform is deployed in production at bloomreach and has been battle tested for large scale load. (millions of documents and hundreds of collections). # Internals: = The internals of the API and performance : http://engineering.bloomreach.com/solrcloud-rebalance-api/ It is built on top of the admin collections API as an action (with various flavors). At a high level, the rebalance api provides 2 constructs: Scaling Strategy: Decides how to move the data. Every flavor has multiple options which can be reviewed in the api spec. Re-distribute - Move around data in the cluster based on capacity/allocation. Auto Shard - Dynamically shard a collection to any size. Smart Merge - Distributed Mode - Helps merging data from a larger shard setup into smaller one. (the source should be divisible by destination) Scale up - Add replicas on the fly Scale Down - Remove replicas on the fly Allocation Strategy: Decides where to put the data. (Nodes with least cores, Nodes that do not have this collection etc). Custom implementations can be built on top as well. One other example is Availability Zone aware. Distribute data such that every replica is placed on different availability zone to support HA. # Detailed API Spec: https://github.com/bloomreach/solrcloud-rebalance-api # Contributors: = Nitin Sharma Suruchi Shah # Questions/Comments: = You can reach me at nitin.sha...@bloomreach.com > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL:
[jira] [Updated] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9241: --- Description: This is the v1 of the patch for Solrcloud Rebalance api (as described in http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to provide a zero downtime mechanism to perform data manipulation and efficient core allocation in solrcloud. This API was envisioned to be the base layer that enables Solrcloud to be an auto scaling platform. (and work in unison with other complementing monitoring and scaling features). Patch Status: === The patch is work in progress and incremental. We have done a few rounds of code clean up. We wanted to get the patch going first to get initial feed back. We will continue to work on making it more open source friendly and easily testable. Deployment Status: The platform is deployed in production at bloomreach and has been battle tested for large scale load. (millions of documents and hundreds of collections). Internals: = The internals of the API and performance : http://engineering.bloomreach.com/solrcloud-rebalance-api/ It is built on top of the admin collections API as an action (with various flavors). At a high level, the rebalance api provides 2 constructs: Scaling Strategy: Decides how to move the data. Every flavor has multiple options which can be reviewed in the api spec. Re-distribute - Move around data in the cluster based on capacity/allocation. Auto Shard - Dynamically shard a collection to any size. Smart Merge - Distributed Mode - Helps merging data from a larger shard setup into smaller one. (the source should be divisible by destination) Scale up - Add replicas on the fly Scale Down - Remove replicas on the fly Allocation Strategy: Decides where to put the data. (Nodes with least cores, Nodes that do not have this collection etc). Custom implementations can be built on top as well. One other example is Availability Zone aware. Distribute data such that every replica is placed on different availability zone to support HA. Detailed API Spec: https://github.com/bloomreach/solrcloud-rebalance-api Contributors: = Nitin Sharma Suruchi Shah Questions/Comments: = You can reach me at nitin.sha...@bloomreach.com was: This is the v1 of the patch for Solrcloud Rebalance api (as described in http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to provide a zero downtime mechanism to perform data manipulation and efficient core allocation in solrcloud. This API was envisioned to be the base layer that enables Solrcloud to be an auto scaling platform. (and work in unison with other complementing monitoring and scaling features). Patch Status: === The patch is work in progress and incremental. We have done a few rounds of code clean up. We wanted to get the patch going first to get initial feed back. We will continue to work on making it more open source friendly and easily testable. Deployment Status: The platform is deployed in production at bloomreach and has been battle tested for large scale load. (millions of documents and hundreds of collections). Internals: = The internals of the API and performance : http://engineering.bloomreach.com/solrcloud-rebalance-api/ It is built on top of the admin collections API as an action (with various flavors). At a high level, the rebalance api provides 2 constructs: Scaling Strategy: Decides how to move the data. Every flavor has multiple options which can be reviewed in the api spec. Re-distribute - Move around data in the cluster based on capacity/allocation. Auto Shard - Dynamically shard a collection to any size. Smart Merge - Distributed Mode - Helps merging data from a larger shard setup into smaller one. (the source should be divisible by destination) Scale up - Add replicas on the fly Scale Down - Remove replicas on the fly Allocation Strategy: Decides where to put the data. (Nodes with least cores, Nodes that do not have this collection etc). Custom implementations can be built on top as well. One other example is Availability Zone aware. Distribute data such that every replica is placed on different availability zone to support HA. # Detailed API Spec: https://github.com/bloomreach/solrcloud-rebalance-api # Contributors: = Nitin Sharma Suruchi Shah # Questions/Comments: = You can reach me at nitin.sha...@bloomreach.com > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL:
[jira] [Updated] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9241: --- Description: This is the v1 of the patch for Solrcloud Rebalance api (as described in http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to provide a zero downtime mechanism to perform data manipulation and efficient core allocation in solrcloud. This API was envisioned to be the base layer that enables Solrcloud to be an auto scaling platform. (and work in unison with other complementing monitoring and scaling features). # Patch Status: === The patch is work in progress and incremental. We have done a few rounds of code clean up. We wanted to get the patch going first to get initial feed back. We will continue to work on making it more open source friendly and easily testable. # Deployment Status: The platform is deployed in production at bloomreach and has been battle tested for large scale load. (millions of documents and hundreds of collections). # Internals: = The internals of the API and performance : http://engineering.bloomreach.com/solrcloud-rebalance-api/ It is built on top of the admin collections API as an action (with various flavors). At a high level, the rebalance api provides 2 constructs: Scaling Strategy: Decides how to move the data. Every flavor has multiple options which can be reviewed in the api spec. Re-distribute - Move around data in the cluster based on capacity/allocation. Auto Shard - Dynamically shard a collection to any size. Smart Merge - Distributed Mode - Helps merging data from a larger shard setup into smaller one. (the source should be divisible by destination) Scale up - Add replicas on the fly Scale Down - Remove replicas on the fly Allocation Strategy: Decides where to put the data. (Nodes with least cores, Nodes that do not have this collection etc). Custom implementations can be built on top as well. One other example is Availability Zone aware. Distribute data such that every replica is placed on different availability zone to support HA. # Detailed API Spec: https://github.com/bloomreach/solrcloud-rebalance-api # Contributors: = Nitin Sharma Suruchi Shah # Questions/Comments: = You can reach me at nitin.sha...@bloomreach.com was: This is the v1 of the patch for Solrcloud Rebalance api, built at Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to provide a zero downtime mechanism to perform data manipulation and efficient core allocation in solrcloud. This API was envisioned to be the base layer that enables Solrcloud to be an auto scaling platform. (and work in unison with other complementing monitoring and scaling features). # Patch Status: === The patch is work in progress and incremental. We have done a few rounds of code clean up. We wanted to get the patch going first to get initial feed back. We will continue to work on making it more open source friendly and easily testable. # Deployment Status: The platform is deployed in production at bloomreach and has been battle tested for large scale load. (millions of documents and hundreds of collections). # Internals: = The internals of the API and performance : http://engineering.bloomreach.com/solrcloud-rebalance-api/ It is built on top of the admin collections API as an action (with various flavors). At a high level, the rebalance api provides 2 constructs: Scaling Strategy: Decides how to move the data. Every flavor has multiple options which can be reviewed in the api spec. Re-distribute - Move around data in the cluster based on capacity/allocation. Auto Shard - Dynamically shard a collection to any size. Smart Merge - Distributed Mode - Helps merging data from a larger shard setup into smaller one. (the source should be divisible by destination) Scale up - Add replicas on the fly Scale Down - Remove replicas on the fly Allocation Strategy: Decides where to put the data. (Nodes with least cores, Nodes that do not have this collection etc). Custom implementations can be built on top as well. One other example is Availability Zone aware. Distribute data such that every replica is placed on different availability zone to support HA. # Detailed API Spec: https://github.com/bloomreach/solrcloud-rebalance-api # Contributors: = Nitin Sharma Suruchi Shah # Questions/Comments: = You can reach me at nitin.sha...@bloomreach.com > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project:
[jira] [Created] (SOLR-9241) Rebalance API for SolrCloud
Nitin Sharma created SOLR-9241: -- Summary: Rebalance API for SolrCloud Key: SOLR-9241 URL: https://issues.apache.org/jira/browse/SOLR-9241 Project: Solr Issue Type: New Feature Components: SolrCloud Affects Versions: 4.6.1 Environment: Ubuntu, Mac OsX Reporter: Nitin Sharma Fix For: 4.6.1 Attachments: rebalance.diff This is the v1 of the patch for Solrcloud Rebalance api, built at Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API is to provide a zero downtime mechanism to perform data manipulation and efficient core allocation in solrcloud. This API was envisioned to be the base layer that enables Solrcloud to be an auto scaling platform. (and work in unison with other complementing monitoring and scaling features). # Patch Status: === The patch is work in progress and incremental. We have done a few rounds of code clean up. We wanted to get the patch going first to get initial feed back. We will continue to work on making it more open source friendly and easily testable. # Deployment Status: The platform is deployed in production at bloomreach and has been battle tested for large scale load. (millions of documents and hundreds of collections). # Internals: = The internals of the API and performance : http://engineering.bloomreach.com/solrcloud-rebalance-api/ It is built on top of the admin collections API as an action (with various flavors). At a high level, the rebalance api provides 2 constructs: Scaling Strategy: Decides how to move the data. Every flavor has multiple options which can be reviewed in the api spec. Re-distribute - Move around data in the cluster based on capacity/allocation. Auto Shard - Dynamically shard a collection to any size. Smart Merge - Distributed Mode - Helps merging data from a larger shard setup into smaller one. (the source should be divisible by destination) Scale up - Add replicas on the fly Scale Down - Remove replicas on the fly Allocation Strategy: Decides where to put the data. (Nodes with least cores, Nodes that do not have this collection etc). Custom implementations can be built on top as well. One other example is Availability Zone aware. Distribute data such that every replica is placed on different availability zone to support HA. # Detailed API Spec: https://github.com/bloomreach/solrcloud-rebalance-api # Contributors: = Nitin Sharma Suruchi Shah # Questions/Comments: = You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9241) Rebalance API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitin Sharma updated SOLR-9241: --- Attachment: rebalance.diff Rebalance API for SolrCloud > Rebalance API for SolrCloud > --- > > Key: SOLR-9241 > URL: https://issues.apache.org/jira/browse/SOLR-9241 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: 4.6.1 > Environment: Ubuntu, Mac OsX >Reporter: Nitin Sharma > Labels: Cluster, SolrCloud > Fix For: 4.6.1 > > Attachments: rebalance.diff > > Original Estimate: 2,016h > Remaining Estimate: 2,016h > > This is the v1 of the patch for Solrcloud Rebalance api, built at Bloomreach > by Nitin Sharma and Suruchi Shah. The goal of the API is to provide a zero > downtime mechanism to perform data manipulation and efficient core > allocation in solrcloud. This API was envisioned to be the base layer that > enables Solrcloud to be an auto scaling platform. (and work in unison with > other complementing monitoring and scaling features). > # Patch Status: > === > The patch is work in progress and incremental. We have done a few rounds of > code clean up. We wanted to get the patch going first to get initial feed > back. We will continue to work on making it more open source friendly and > easily testable. > # Deployment Status: > > The platform is deployed in production at bloomreach and has been battle > tested for large scale load. (millions of documents and hundreds of > collections). > # Internals: > = > The internals of the API and performance : > http://engineering.bloomreach.com/solrcloud-rebalance-api/ > It is built on top of the admin collections API as an action (with various > flavors). At a high level, the rebalance api provides 2 constructs: > Scaling Strategy: Decides how to move the data. Every flavor has multiple > options which can be reviewed in the api spec. > Re-distribute - Move around data in the cluster based on capacity/allocation. > Auto Shard - Dynamically shard a collection to any size. > Smart Merge - Distributed Mode - Helps merging data from a larger shard setup > into smaller one. (the source should be divisible by destination) > Scale up - Add replicas on the fly > Scale Down - Remove replicas on the fly > Allocation Strategy: Decides where to put the data. (Nodes with least > cores, Nodes that do not have this collection etc). Custom implementations > can be built on top as well. One other example is Availability Zone aware. > Distribute data such that every replica is placed on different availability > zone to support HA. > # Detailed API Spec: > > https://github.com/bloomreach/solrcloud-rebalance-api > # Contributors: > = > Nitin Sharma > Suruchi Shah > # Questions/Comments: > = > You can reach me at nitin.sha...@bloomreach.com -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3274) ZooKeeper related SolrCloud problems
[
https://issues.apache.org/jira/browse/SOLR-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153691#comment-15153691
]
Nitin Sharma commented on SOLR-3274:
I also ran into this issue with solr 4.6.1. The main issue is that updates go
through fine in solr but solr-zk interactions are broken and few shards are
marked as down. They never recover/reconnect to zk unless force solr restart. I
even tried bumping up zk ensemble to 7 and upto 9 nodes but did not help.
Any suggestions on how to deal with this?
> ZooKeeper related SolrCloud problems
>
>
> Key: SOLR-3274
> URL: https://issues.apache.org/jira/browse/SOLR-3274
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Affects Versions: 4.0-ALPHA
> Environment: Any
>Reporter: Per Steffensen
>
> Same setup as in SOLR-3273. Well if I have to tell the entire truth we have 7
> Solr servers, running 28 slices of the same collection (collA) - all slices
> have one replica (two shards all in all - leader + replica) - 56 cores all in
> all (8 shards on each solr instance). But anyways...
> Besides the problem reported in SOLR-3273, the system seems to run fine under
> high load for several hours, but eventually errors like the ones shown below
> start to occur. I might be wrong, but they all seem to indicate some kind of
> unstability in the collaboration between Solr and ZooKeeper. I have to say
> that I havnt been there to check ZooKeeper "at the moment where those
> exception occur", but basically I dont believe the exceptions occur because
> ZooKeeper is not running stable - at least when I go and check ZooKeeper
> through other "channels" (e.g. my eclipse ZK plugin) it is always accepting
> my connection and generally seems to be doing fine.
> Exception 1) Often the first error we see in solr.log is something like this
> {code}
> Mar 22, 2012 5:06:43 AM org.apache.solr.common.SolrException log
> SEVERE: org.apache.solr.common.SolrException: Cannot talk to ZooKeeper -
> Updates are disabled.
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.zkCheck(DistributedUpdateProcessor.java:678)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:250)
> at org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:140)
> at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:80)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:59)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1540)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:407)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:256)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> {code}
> I believe this error basically occurs because SolrZkClient.isConnected
> reports false, which means that its internal "keeper.getState" does not
> return ZooKeeper.States.CONNECTED. Im pretty sure that it has been CONNECTED
> for a long time, since this error starts occuring after several hours of
> processing without this problem showing. But why is it suddenly not connected
> anymore?!
> Exception
[jira] [Created] (SOLR-7479) Updating Solr-Core Properties on the Fly to avoid core/collection reloads
Nitin Sharma created SOLR-7479: -- Summary: Updating Solr-Core Properties on the Fly to avoid core/collection reloads Key: SOLR-7479 URL: https://issues.apache.org/jira/browse/SOLR-7479 Project: Solr Issue Type: Improvement Components: SolrCloud Affects Versions: 4.6.1 Environment: Any Platform Reporter: Nitin Sharma Currently everytime a property/value is changed in solrcore.properties file, a core/collection reload is needed to pick up the new values. Core/Collection reloads for large collections (example 100 shards) is very expensive (performance wise) and can pose a threat to the collection stability (sometimes the reload fails since the timeout is only 60 seconds).For a RT serving infrastructure, this does not help Would adding a Real Time config api (map) inside solrcloud help? Every solr core can pick up the core specific configs from this Shared Map. This can help with dynamically changing properties without core reloads. If there is a better solution than that, please let me know -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
