[jira] [Commented] (SOLR-11990) Make it possible to co-locate replicas of multiple collections together in a node using policy
[ https://issues.apache.org/jira/browse/SOLR-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546097#comment-16546097 ] Shalin Shekhar Mangar commented on SOLR-11990: -- This patch: # Changes the approach to use a new ConditionType called WITH_COLLECTION that ensures that we project add replica correctly i.e. during computation, adding a replica for the main collection also adds a replica for the with collection so that we don't generate bad suggestions at all # SolrClientNodeStateProvider is aware of the `withCollection` flag and returns them in the state # Move replica, delete collection APIs are aware of with collection # New integration tests in TestWithCollection.testNodeAdded # Still a few nocommits left > Make it possible to co-locate replicas of multiple collections together in a > node using policy > -- > > Key: SOLR-11990 > URL: https://issues.apache.org/jira/browse/SOLR-11990 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling, SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar >Priority: Major > Fix For: master (8.0), 7.5 > > Attachments: SOLR-11990.patch, SOLR-11990.patch, SOLR-11990.patch, > SOLR-11990.patch, SOLR-11990.patch, SOLR-11990.patch > > > It is necessary to co-locate replicas of different collection together in a > node when cross-collection joins are performed. The policy rules framework > should support this use-case. > Example: Co-locate exactly 1 replica of collection A in each node where a > replica of collection B is present. > {code} > {"replica":">0", "collection":"A", "shard":"#EACH", "withCollection":"B"} > {code} > This requires changing create collection, create shard and add replica APIs > as well because we want a replica of collection A to be created first before > a replica of collection B is created so that join queries etc are always > possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11990) Make it possible to co-locate replicas of multiple collections together in a node using policy
[ https://issues.apache.org/jira/browse/SOLR-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523720#comment-16523720 ] Shalin Shekhar Mangar commented on SOLR-11990: -- Patch updated to master after the commits on SOLR-12506 and SOLR-12507. > Make it possible to co-locate replicas of multiple collections together in a > node using policy > -- > > Key: SOLR-11990 > URL: https://issues.apache.org/jira/browse/SOLR-11990 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling, SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-11990.patch, SOLR-11990.patch, SOLR-11990.patch, > SOLR-11990.patch, SOLR-11990.patch > > > It is necessary to co-locate replicas of different collection together in a > node when cross-collection joins are performed. The policy rules framework > should support this use-case. > Example: Co-locate exactly 1 replica of collection A in each node where a > replica of collection B is present. > {code} > {"replica":">0", "collection":"A", "shard":"#EACH", "withCollection":"B"} > {code} > This requires changing create collection, create shard and add replica APIs > as well because we want a replica of collection A to be created first before > a replica of collection B is created so that join queries etc are always > possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11990) Make it possible to co-locate replicas of multiple collections together in a node using policy
[ https://issues.apache.org/jira/browse/SOLR-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517019#comment-16517019 ] Shalin Shekhar Mangar commented on SOLR-11990: -- New patch that removes the WITH_COLLECTION hint, further reducing the amount of changes needed. Tests have been added for create collection, add replica and move replica operations. There are a few changes here that I will extracted to their own issues and commit independently e.g. solrj support for modify collection. > Make it possible to co-locate replicas of multiple collections together in a > node using policy > -- > > Key: SOLR-11990 > URL: https://issues.apache.org/jira/browse/SOLR-11990 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling, SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-11990.patch, SOLR-11990.patch, SOLR-11990.patch, > SOLR-11990.patch > > > It is necessary to co-locate replicas of different collection together in a > node when cross-collection joins are performed. The policy rules framework > should support this use-case. > Example: Co-locate exactly 1 replica of collection A in each node where a > replica of collection B is present. > {code} > {"replica":">0", "collection":"A", "shard":"#EACH", "withCollection":"B"} > {code} > This requires changing create collection, create shard and add replica APIs > as well because we want a replica of collection A to be created first before > a replica of collection B is created so that join queries etc are always > possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11990) Make it possible to co-locate replicas of multiple collections together in a node using policy
[ https://issues.apache.org/jira/browse/SOLR-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460096#comment-16460096 ] Shalin Shekhar Mangar commented on SOLR-11990: -- I've attached another approach which does not modify the policy framework directly but instead modifies Suggesters and Collection APIs to maintain the co-location of two collections. This is still a work in progress and has many nocommits. > Make it possible to co-locate replicas of multiple collections together in a > node using policy > -- > > Key: SOLR-11990 > URL: https://issues.apache.org/jira/browse/SOLR-11990 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling, SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-11990.patch, SOLR-11990.patch, SOLR-11990.patch > > > It is necessary to co-locate replicas of different collection together in a > node when cross-collection joins are performed. The policy rules framework > should support this use-case. > Example: Co-locate exactly 1 replica of collection A in each node where a > replica of collection B is present. > {code} > {"replica":">0", "collection":"A", "shard":"#EACH", "withCollection":"B"} > {code} > This requires changing create collection, create shard and add replica APIs > as well because we want a replica of collection A to be created first before > a replica of collection B is created so that join queries etc are always > possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11990) Make it possible to co-locate replicas of multiple collections together in a node using policy
[ https://issues.apache.org/jira/browse/SOLR-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442366#comment-16442366 ] Noble Paul commented on SOLR-11990: --- The ideal syntax for such a feature would be {code:java} //place all replicas of collection 'comment' where collection 'article' is present {"replica" :"#ALL", "collection" : "comment", "withCollection" :"article" } {code} With this feature * we expect the collection {{"article"}} to be created first. If {{"article"}} is not present creation of collection {{"comment"}} must fail * The collection {{"article"}} must have only one shard * when {{"comments"}} collection is created, it must first identify nodes where replicas are going to be created, and then it must go on and create replicas of {{"article"}} in all those nodes * When a new replica (or a shard) is created for {{"comment"}} , it must do the same as create collection I seriously think it is a lot of work . We will have to change the way the collection APIs work. The way the rules are computed in the Policy framework also needs to change specifically for this use case. The APIs are not written to handle multiple collections simultaneously. I think it will be much easier if we just make this as a collection attribute and deal it specially there. That way, we don't have to touch the Policy framework at all > Make it possible to co-locate replicas of multiple collections together in a > node using policy > -- > > Key: SOLR-11990 > URL: https://issues.apache.org/jira/browse/SOLR-11990 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling, SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-11990.patch, SOLR-11990.patch > > > It is necessary to co-locate replicas of different collection together in a > node when cross-collection joins are performed. The policy rules framework > should support this use-case. > Example: Co-locate exactly 1 replica of collection A in each node where a > replica of collection B is present. > {code} > {"replica":">0", "collection":"A", "shard":"#EACH", "withCollection":"B"} > {code} > This requires changing create collection, create shard and add replica APIs > as well because we want a replica of collection A to be created first before > a replica of collection B is created so that join queries etc are always > possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11990) Make it possible to co-locate replicas of multiple collections together in a node using policy
[ https://issues.apache.org/jira/browse/SOLR-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440633#comment-16440633 ] Shalin Shekhar Mangar commented on SOLR-11990: -- New patch removes broken support for multi-shard `withCollection`. It is not possible to support such a feature today because of ambiguity about how to co-locate multi-shard collections together. We can tackle that in a later issue. The goal behind this issue is to be support joining collections which usually means co-locating a multi-shard `collection` with a single-shard `withCollection` > Make it possible to co-locate replicas of multiple collections together in a > node using policy > -- > > Key: SOLR-11990 > URL: https://issues.apache.org/jira/browse/SOLR-11990 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling, SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-11990.patch, SOLR-11990.patch > > > It is necessary to co-locate replicas of different collection together in a > node when cross-collection joins are performed. The policy rules framework > should support this use-case. > Example: Co-locate exactly 1 replica of collection A in each node where a > replica of collection B is present. > {code} > {"replica":1, "collection":"A", "shard":"#EACH", "withCollection":"B"} > {code} > This requires changing create collection, create shard and add replica APIs > as well because we want a replica of collection A to be created first before > a replica of collection B is created so that join queries etc are always > possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11990) Make it possible to co-locate replicas of multiple collections together in a node using policy
[ https://issues.apache.org/jira/browse/SOLR-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439935#comment-16439935 ] Shalin Shekhar Mangar commented on SOLR-11990: -- First cut of the feature. It only supports `withCollection` for collections with a single shard because we do not have access to the other clauses (specifically the collection and shard values) at the time of the Operand.match operation. There is still a lot of work and tests left: # Collection APIs need to be fixed to pre-create `withCollection` replicas before replicas of `collection` can be created. # Delete `withCollection` replicas if `collection` replicas are moved or deleted > Make it possible to co-locate replicas of multiple collections together in a > node using policy > -- > > Key: SOLR-11990 > URL: https://issues.apache.org/jira/browse/SOLR-11990 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling, SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-11990.patch > > > It is necessary to co-locate replicas of different collection together in a > node when cross-collection joins are performed. The policy rules framework > should support this use-case. > Example: Co-locate exactly 1 replica of collection A in each node where a > replica of collection B is present. > {code} > {"replica":1, "collection":"A", "shard":"#EACH", "withCollection":"B"} > {code} > This requires changing create collection, create shard and add replica APIs > as well because we want a replica of collection A to be created first before > a replica of collection B is created so that join queries etc are always > possible. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org