[jira] [Commented] (SOLR-11990) Make it possible to co-locate replicas of multiple collections together in a node using policy
[
https://issues.apache.org/jira/browse/SOLR-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546097#comment-16546097
]
Shalin Shekhar Mangar commented on SOLR-11990:
--
This patch:
# Changes the approach to use a new ConditionType called WITH_COLLECTION that
ensures that we project add replica correctly i.e. during computation, adding a
replica for the main collection also adds a replica for the with collection so
that we don't generate bad suggestions at all
# SolrClientNodeStateProvider is aware of the `withCollection` flag and returns
them in the state
# Move replica, delete collection APIs are aware of with collection
# New integration tests in TestWithCollection.testNodeAdded
# Still a few nocommits left
> Make it possible to co-locate replicas of multiple collections together in a
> node using policy
> --
>
> Key: SOLR-11990
> URL: https://issues.apache.org/jira/browse/SOLR-11990
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling, SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: master (8.0), 7.5
>
> Attachments: SOLR-11990.patch, SOLR-11990.patch, SOLR-11990.patch,
> SOLR-11990.patch, SOLR-11990.patch, SOLR-11990.patch
>
>
> It is necessary to co-locate replicas of different collection together in a
> node when cross-collection joins are performed. The policy rules framework
> should support this use-case.
> Example: Co-locate exactly 1 replica of collection A in each node where a
> replica of collection B is present.
> {code}
> {"replica":">0", "collection":"A", "shard":"#EACH", "withCollection":"B"}
> {code}
> This requires changing create collection, create shard and add replica APIs
> as well because we want a replica of collection A to be created first before
> a replica of collection B is created so that join queries etc are always
> possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11990) Make it possible to co-locate replicas of multiple collections together in a node using policy
[
https://issues.apache.org/jira/browse/SOLR-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16523720#comment-16523720
]
Shalin Shekhar Mangar commented on SOLR-11990:
--
Patch updated to master after the commits on SOLR-12506 and SOLR-12507.
> Make it possible to co-locate replicas of multiple collections together in a
> node using policy
> --
>
> Key: SOLR-11990
> URL: https://issues.apache.org/jira/browse/SOLR-11990
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling, SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: SOLR-11990.patch, SOLR-11990.patch, SOLR-11990.patch,
> SOLR-11990.patch, SOLR-11990.patch
>
>
> It is necessary to co-locate replicas of different collection together in a
> node when cross-collection joins are performed. The policy rules framework
> should support this use-case.
> Example: Co-locate exactly 1 replica of collection A in each node where a
> replica of collection B is present.
> {code}
> {"replica":">0", "collection":"A", "shard":"#EACH", "withCollection":"B"}
> {code}
> This requires changing create collection, create shard and add replica APIs
> as well because we want a replica of collection A to be created first before
> a replica of collection B is created so that join queries etc are always
> possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11990) Make it possible to co-locate replicas of multiple collections together in a node using policy
[
https://issues.apache.org/jira/browse/SOLR-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517019#comment-16517019
]
Shalin Shekhar Mangar commented on SOLR-11990:
--
New patch that removes the WITH_COLLECTION hint, further reducing the amount of
changes needed. Tests have been added for create collection, add replica and
move replica operations. There are a few changes here that I will extracted to
their own issues and commit independently e.g. solrj support for modify
collection.
> Make it possible to co-locate replicas of multiple collections together in a
> node using policy
> --
>
> Key: SOLR-11990
> URL: https://issues.apache.org/jira/browse/SOLR-11990
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling, SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: SOLR-11990.patch, SOLR-11990.patch, SOLR-11990.patch,
> SOLR-11990.patch
>
>
> It is necessary to co-locate replicas of different collection together in a
> node when cross-collection joins are performed. The policy rules framework
> should support this use-case.
> Example: Co-locate exactly 1 replica of collection A in each node where a
> replica of collection B is present.
> {code}
> {"replica":">0", "collection":"A", "shard":"#EACH", "withCollection":"B"}
> {code}
> This requires changing create collection, create shard and add replica APIs
> as well because we want a replica of collection A to be created first before
> a replica of collection B is created so that join queries etc are always
> possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11990) Make it possible to co-locate replicas of multiple collections together in a node using policy
[
https://issues.apache.org/jira/browse/SOLR-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460096#comment-16460096
]
Shalin Shekhar Mangar commented on SOLR-11990:
--
I've attached another approach which does not modify the policy framework
directly but instead modifies Suggesters and Collection APIs to maintain the
co-location of two collections. This is still a work in progress and has many
nocommits.
> Make it possible to co-locate replicas of multiple collections together in a
> node using policy
> --
>
> Key: SOLR-11990
> URL: https://issues.apache.org/jira/browse/SOLR-11990
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling, SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: SOLR-11990.patch, SOLR-11990.patch, SOLR-11990.patch
>
>
> It is necessary to co-locate replicas of different collection together in a
> node when cross-collection joins are performed. The policy rules framework
> should support this use-case.
> Example: Co-locate exactly 1 replica of collection A in each node where a
> replica of collection B is present.
> {code}
> {"replica":">0", "collection":"A", "shard":"#EACH", "withCollection":"B"}
> {code}
> This requires changing create collection, create shard and add replica APIs
> as well because we want a replica of collection A to be created first before
> a replica of collection B is created so that join queries etc are always
> possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11990) Make it possible to co-locate replicas of multiple collections together in a node using policy
[
https://issues.apache.org/jira/browse/SOLR-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442366#comment-16442366
]
Noble Paul commented on SOLR-11990:
---
The ideal syntax for such a feature would be
{code:java}
//place all replicas of collection 'comment' where collection 'article' is
present
{"replica" :"#ALL", "collection" : "comment", "withCollection" :"article" }
{code}
With this feature
* we expect the collection {{"article"}} to be created first. If {{"article"}}
is not present creation of collection {{"comment"}} must fail
* The collection {{"article"}} must have only one shard
* when {{"comments"}} collection is created, it must first identify nodes
where replicas are going to be created, and then it must go on and create
replicas of {{"article"}} in all those nodes
* When a new replica (or a shard) is created for {{"comment"}} , it must do
the same as create collection
I seriously think it is a lot of work . We will have to change the way the
collection APIs work. The way the rules are computed in the Policy framework
also needs to change specifically for this use case. The APIs are not written
to handle multiple collections simultaneously. I think it will be much easier
if we just make this as a collection attribute and deal it specially there.
That way, we don't have to touch the Policy framework at all
> Make it possible to co-locate replicas of multiple collections together in a
> node using policy
> --
>
> Key: SOLR-11990
> URL: https://issues.apache.org/jira/browse/SOLR-11990
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling, SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: SOLR-11990.patch, SOLR-11990.patch
>
>
> It is necessary to co-locate replicas of different collection together in a
> node when cross-collection joins are performed. The policy rules framework
> should support this use-case.
> Example: Co-locate exactly 1 replica of collection A in each node where a
> replica of collection B is present.
> {code}
> {"replica":">0", "collection":"A", "shard":"#EACH", "withCollection":"B"}
> {code}
> This requires changing create collection, create shard and add replica APIs
> as well because we want a replica of collection A to be created first before
> a replica of collection B is created so that join queries etc are always
> possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11990) Make it possible to co-locate replicas of multiple collections together in a node using policy
[
https://issues.apache.org/jira/browse/SOLR-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440633#comment-16440633
]
Shalin Shekhar Mangar commented on SOLR-11990:
--
New patch removes broken support for multi-shard `withCollection`. It is not
possible to support such a feature today because of ambiguity about how to
co-locate multi-shard collections together. We can tackle that in a later
issue. The goal behind this issue is to be support joining collections which
usually means co-locating a multi-shard `collection` with a single-shard
`withCollection`
> Make it possible to co-locate replicas of multiple collections together in a
> node using policy
> --
>
> Key: SOLR-11990
> URL: https://issues.apache.org/jira/browse/SOLR-11990
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling, SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: SOLR-11990.patch, SOLR-11990.patch
>
>
> It is necessary to co-locate replicas of different collection together in a
> node when cross-collection joins are performed. The policy rules framework
> should support this use-case.
> Example: Co-locate exactly 1 replica of collection A in each node where a
> replica of collection B is present.
> {code}
> {"replica":1, "collection":"A", "shard":"#EACH", "withCollection":"B"}
> {code}
> This requires changing create collection, create shard and add replica APIs
> as well because we want a replica of collection A to be created first before
> a replica of collection B is created so that join queries etc are always
> possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11990) Make it possible to co-locate replicas of multiple collections together in a node using policy
[
https://issues.apache.org/jira/browse/SOLR-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439935#comment-16439935
]
Shalin Shekhar Mangar commented on SOLR-11990:
--
First cut of the feature. It only supports `withCollection` for collections
with a single shard because we do not have access to the other clauses
(specifically the collection and shard values) at the time of the Operand.match
operation. There is still a lot of work and tests left:
# Collection APIs need to be fixed to pre-create `withCollection` replicas
before replicas of `collection` can be created.
# Delete `withCollection` replicas if `collection` replicas are moved or deleted
> Make it possible to co-locate replicas of multiple collections together in a
> node using policy
> --
>
> Key: SOLR-11990
> URL: https://issues.apache.org/jira/browse/SOLR-11990
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling, SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: SOLR-11990.patch
>
>
> It is necessary to co-locate replicas of different collection together in a
> node when cross-collection joins are performed. The policy rules framework
> should support this use-case.
> Example: Co-locate exactly 1 replica of collection A in each node where a
> replica of collection B is present.
> {code}
> {"replica":1, "collection":"A", "shard":"#EACH", "withCollection":"B"}
> {code}
> This requires changing create collection, create shard and add replica APIs
> as well because we want a replica of collection A to be created first before
> a replica of collection B is created so that join queries etc are always
> possible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
