[jira] [Comment Edited] (LUCENE-3759) Support joining in a distributed environment.
[
https://issues.apache.org/jira/browse/LUCENE-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571146#comment-16571146
]
jyoti Tiwari edited comment on LUCENE-3759 at 8/7/18 5:59 AM:
--
Hello Ashish Dutta,
Please look into my issue which i am facing on solr4 while making join query
across sharded collection on same node.
Solr4 on cloud joining across two sharded core i.e
engineeringlogs_shard1_replica1 on machine 1 and micrologs_shard1_replica1 on
machine 1
machine1 - engineeringlogs_shard1_replica1 (A), micrologs_shard1_replica1(B)
machine 2- engineeringlogs_shard2_replica1(A1) , micrologs_shard2_replica1(B1)
query time join on engineeringlogs_shard1_replica1 (A):
fq: "\{!join from=Log_type to=Log_type
fromIndex=micrologs_shard1_replica1}SerialNumber:\"000123456789\""
want to perform join across A and A1 on same machine 1,but it is not working
fine. error is:
"error": \{ "metadata": [ "error-class",
"org.apache.solr.common.SolrException", "root-error-class",
"org.apache.solr.common.SolrException" ], "msg": "Cross-core join: no such core
micrologs_shard1_replica1", "code": 400 }
Please guide me how should i proceed so that this join query will work fine for
sharded collection for solr4 cloud.
was (Author: jyoti609):
Solr4 on cloud joining across two sharded core i.e
engineeringlogs_shard1_replica1 on machine 1 and micrologs_shard1_replica1 on
machine 1
machine1 - engineeringlogs_shard1_replica1 (A), micrologs_shard1_replica1(B)
machine 2- engineeringlogs_shard2_replica1(A1) , micrologs_shard2_replica1(B1)
query time join on engineeringlogs_shard1_replica1 (A):
fq: "\{!join from=Log_type to=Log_type
fromIndex=micrologs_shard1_replica1}SerialNumber:\"000123456789\""
want to perform join across A and A1 on same machine 1,but it is not working
fine. error is:
"error": \{ "metadata": [ "error-class",
"org.apache.solr.common.SolrException", "root-error-class",
"org.apache.solr.common.SolrException" ], "msg": "Cross-core join: no such core
micrologs_shard1_replica1", "code": 400 }
Please guide me how should i proceed so that this join query will work fine for
sharded collection for solr4 cloud.
> Support joining in a distributed environment.
> -
>
> Key: LUCENE-3759
> URL: https://issues.apache.org/jira/browse/LUCENE-3759
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/join
>Reporter: Martijn van Groningen
>Priority: Major
>
> Add two more methods in JoinUtil to support joining in a distributed manner.
> * Method to retrieve all from values.
> * Method to create a TermsQuery based on a set of from terms.
> With these two methods distributed joining can be supported following these
> steps:
> # Retrieve from values from each shard
> # Merge the retrieved from values.
> # Create a TermsQuery based on the merged from terms and send this query to
> all shards.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3759) Support joining in a distributed environment.
[
https://issues.apache.org/jira/browse/LUCENE-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571148#comment-16571148
]
jyoti Tiwari commented on LUCENE-3759:
--
Hello Ashish Dutta,
Please look into my issue which i am facing on solr4 while making join query
across sharded collection on same node.
Solr4 on cloud joining across two sharded core i.e
engineeringlogs_shard1_replica1 on machine 1 and micrologs_shard1_replica1 on
machine 1
machine1 - engineeringlogs_shard1_replica1 (A), micrologs_shard1_replica1(B)
machine 2- engineeringlogs_shard2_replica1(A1) , micrologs_shard2_replica1(B1)
query time join on engineeringlogs_shard1_replica1 (A):
fq: "{!join from=Log_type to=Log_type
fromIndex=micrologs_shard1_replica1}SerialNumber:\"000123456789\""
want to perform join across A and A1 on same machine 1,but it is not working
fine. error is:
"error": { "metadata": [ "error-class", "org.apache.solr.common.SolrException",
"root-error-class", "org.apache.solr.common.SolrException" ], "msg":
"Cross-core join: no such core micrologs_shard1_replica1", "code": 400 }
Please guide me how should i proceed so that this join query will work fine for
sharded collection for solr4 cloud.
> Support joining in a distributed environment.
> -
>
> Key: LUCENE-3759
> URL: https://issues.apache.org/jira/browse/LUCENE-3759
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/join
>Reporter: Martijn van Groningen
>Priority: Major
>
> Add two more methods in JoinUtil to support joining in a distributed manner.
> * Method to retrieve all from values.
> * Method to create a TermsQuery based on a set of from terms.
> With these two methods distributed joining can be supported following these
> steps:
> # Retrieve from values from each shard
> # Merge the retrieved from values.
> # Create a TermsQuery based on the merged from terms and send this query to
> all shards.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-3759) Support joining in a distributed environment.
[
https://issues.apache.org/jira/browse/LUCENE-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571146#comment-16571146
]
jyoti Tiwari edited comment on LUCENE-3759 at 8/7/18 5:57 AM:
--
Solr4 on cloud joining across two sharded core i.e
engineeringlogs_shard1_replica1 on machine 1 and micrologs_shard1_replica1 on
machine 1
machine1 - engineeringlogs_shard1_replica1 (A), micrologs_shard1_replica1(B)
machine 2- engineeringlogs_shard2_replica1(A1) , micrologs_shard2_replica1(B1)
query time join on engineeringlogs_shard1_replica1 (A):
fq: "\{!join from=Log_type to=Log_type
fromIndex=micrologs_shard1_replica1}SerialNumber:\"000123456789\""
want to perform join across A and A1 on same machine 1,but it is not working
fine. error is:
"error": \{ "metadata": [ "error-class",
"org.apache.solr.common.SolrException", "root-error-class",
"org.apache.solr.common.SolrException" ], "msg": "Cross-core join: no such core
micrologs_shard1_replica1", "code": 400 }
Please guide me how should i proceed so that this join query will work fine for
sharded collection for solr4 cloud.
was (Author: jyoti609):
Solr4 on cloud joining across two sharded core i.e
engineeringlogs_shard1_replica1 on machine 1 and micrologs_shard1_replica1 on
machine 1
machine1 - engineeringlogs_shard1_replica1 (A), micrologs_shard1_replica1(B)
machine 2- engineeringlogs_shard2_replica1(A1) , micrologs_shard2_replica1(B1)
query time join on engineeringlogs_shard1_replica1 (A):
fq: "\{!join from=Log_type to=Log_type
fromIndex=micrologs_shard1_replica1}SerialNumber:\"000123456789\""
want to perform join across A and A1 on same machine 1,but it is not working
fine. error is:
"error": \{ "metadata": [ "error-class",
"org.apache.solr.common.SolrException", "root-error-class",
"org.apache.solr.common.SolrException" ], "msg": "Cross-core join: no such core
micrologs_shard1_replica1", "code": 400 }
Please guide me how should i proceed so that this join query will work fine for
sharded collection for solr4 cloud.
> Support joining in a distributed environment.
> -
>
> Key: LUCENE-3759
> URL: https://issues.apache.org/jira/browse/LUCENE-3759
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/join
>Reporter: Martijn van Groningen
>Priority: Major
>
> Add two more methods in JoinUtil to support joining in a distributed manner.
> * Method to retrieve all from values.
> * Method to create a TermsQuery based on a set of from terms.
> With these two methods distributed joining can be supported following these
> steps:
> # Retrieve from values from each shard
> # Merge the retrieved from values.
> # Create a TermsQuery based on the merged from terms and send this query to
> all shards.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3759) Support joining in a distributed environment.
[
https://issues.apache.org/jira/browse/LUCENE-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571146#comment-16571146
]
jyoti Tiwari commented on LUCENE-3759:
--
Solr4 on cloud joining across two sharded core i.e
engineeringlogs_shard1_replica1 on machine 1 and micrologs_shard1_replica1 on
machine 1
machine1 - engineeringlogs_shard1_replica1 (A), micrologs_shard1_replica1(B)
machine 2- engineeringlogs_shard2_replica1(A1) , micrologs_shard2_replica1(B1)
query time join on engineeringlogs_shard1_replica1 (A):
fq: "\{!join from=Log_type to=Log_type
fromIndex=micrologs_shard1_replica1}SerialNumber:\"000123456789\""
want to perform join across A and A1 on same machine 1,but it is not working
fine. error is:
"error": \{ "metadata": [ "error-class",
"org.apache.solr.common.SolrException", "root-error-class",
"org.apache.solr.common.SolrException" ], "msg": "Cross-core join: no such core
micrologs_shard1_replica1", "code": 400 }
Please guide me how should i proceed so that this join query will work fine for
sharded collection for solr4 cloud.
> Support joining in a distributed environment.
> -
>
> Key: LUCENE-3759
> URL: https://issues.apache.org/jira/browse/LUCENE-3759
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/join
>Reporter: Martijn van Groningen
>Priority: Major
>
> Add two more methods in JoinUtil to support joining in a distributed manner.
> * Method to retrieve all from values.
> * Method to create a TermsQuery based on a set of from terms.
> With these two methods distributed joining can be supported following these
> steps:
> # Retrieve from values from each shard
> # Merge the retrieved from values.
> # Create a TermsQuery based on the merged from terms and send this query to
> all shards.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-8297) Allow join query over 2 sharded collections: enhance functionality and exception handling
[
https://issues.apache.org/jira/browse/SOLR-8297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563802#comment-16563802
]
jyoti Tiwari edited comment on SOLR-8297 at 7/31/18 3:08 PM:
-
Erick:
i guess i am fulfilling all condition to run solr join query given by you in
points A) exception handling and B) functional enhancement ,
details:
solr version: 4
total nodes: 2
total collection: 2, let say engineeringlogs1, engineeringlogs2
total core in one node: engineeringlogs1_shard1_replica1 and
engineeringlogs2_shard1_replica1
total core in 2nd node: engineeringlogs1_shard2_replica1 and
engineeringlogs2_shard2_replica1
trying to join over over one node between two core,
(engineeringlogs1_shard1_replica1 and engineeringlogs2_shard1_replica1)
query:
on one default core: engineeringlogs1_shard1_replica1
{!join from=Maximum_Battery_Charge to=Check_Battery_Charge
fromIndex=engineeringlogs2_shard1_replica1}Initial_Battery_Charge: "87%"
but stil this issue is coming
find below error:
"error": \{ "metadata": [ "error-class",
"org.apache.solr.common.SolrException", "root-error-class",
"org.apache.solr.common.SolrException" ], "msg": "Cross-core join: no such core
engineeringlogs2_shard1_replica1", "code": 400 } }
Please get me a solution how i can resolve this issue.
was (Author: jyoti609):
Erick:
i guess i am fulfilling all condition to run solr join query given by you in
points A) exception handling and B) functional enhancement ,
details:
solr version: 4
total nodes: 2
total collection: 2, let say engineeringlogs1, engineeringlogs2
total core in one node: engineeringlogs1_shard1_replica1 and
engineeringlogs2_shard1_replica1
total core in 2nd node: engineeringlogs1_shard2_replica1 and
engineeringlogs2_shard2_replica1
trying to join over over one node between two core,
(engineeringlogs1_shard1_replica1 and engineeringlogs2_shard1_replica1)
query:
on one default core: engineeringlogs1_shard1_replica1
{!join from=Maximum_Battery_Charge to=Check_Battery_Charge
fromIndex=engineeringlogs2_shard1_replica1}
Initial_Battery_Charge: "87%"
but stil this issue is coming
find below error:
"error": \{ "metadata": [ "error-class",
"org.apache.solr.common.SolrException", "root-error-class",
"org.apache.solr.common.SolrException" ], "msg": "Cross-core join: no such core
engineeringlogs2_shard1_replica1", "code": 400 } }
Please get me a solution how i can resolve this issue.
> Allow join query over 2 sharded collections: enhance functionality and
> exception handling
> -
>
> Key: SOLR-8297
> URL: https://issues.apache.org/jira/browse/SOLR-8297
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
>Affects Versions: 5.3
>Reporter: Paul Blanchaert
>Priority: Major
> Attachments: SOLR-8297.patch, SOLR-8297_Latest.patch
>
>
> h2. Proposal
> h3. General Idea
> Approach [~shikhasomani]'s range check algorithm to the most cases
> h3. Join behavior depending on router types of joined collections
> || to\\from ||CompositeId||Implicit||
> ||CompositeId| shard range check, see table below | allow |
> ||Implicit| allow | shard to shard |
> h3. CompositeId to CompositeId join behaviour for certain number of shards
>
> || to\\from ||single||>1||
> ||single| allow (as is) | allow (range check) |
> ||>1| allow (as is) | per shard range check |
> h3. Rules from the tables above
> * joining from/to CompositeId and Implicit is blindly allowed, it pick ups
> any collocated replica, because users who do that probably understand what
> they do.
> * when both sides are Implicit let's join shards by name. ie if request hits
> collectionTO_shardY_replica2 at a node, the collocated
> collectionFROM_shardY_replica* is expected.
> * when both sides are CompositeId
> ** from single shard to single shard - nobrainer, just needs collocated
> replica;
> ** from multiple shards to single shard - all "from" shards (any it's
> replicas) are picked for joining
> ** from single shard to multiple shards - existing SOLR-4905 functionality
> ** from multiple to multiple - generic range check algorithm
> ### check that fromField and toField are router.keys in these collections
> ### take shard range for the current "to" collection replica (keep in mind
> that request is distributed across "to" collection shards)
> ### enumerate "from" collection shrads, find their subset which covers "to"
> shard range (this allows to handle any number of shards at both sides)
> ### pickup collocated replicas of these "from" shard subset
> h3. Caveat
> this is quite sensitive to shard allocation (and/or replica placement) ie
> failed "from" replica cannot be collocated with the required "to" shard.
> h2. Initial
[jira] [Comment Edited] (SOLR-8297) Allow join query over 2 sharded collections: enhance functionality and exception handling
[
https://issues.apache.org/jira/browse/SOLR-8297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563802#comment-16563802
]
jyoti Tiwari edited comment on SOLR-8297 at 7/31/18 3:06 PM:
-
Erick:
i guess i am fulfilling all condition to run solr join query given by you in
points A) exception handling and B) functional enhancement ,
details:
solr version: 4
total nodes: 2
total collection: 2, let say engineeringlogs1, engineeringlogs2
total core in one node: engineeringlogs1_shard1_replica1 and
engineeringlogs2_shard1_replica1
total core in 2nd node: engineeringlogs1_shard2_replica1 and
engineeringlogs2_shard2_replica1
trying to join over over one node between two core,
(engineeringlogs1_shard1_replica1 and engineeringlogs2_shard1_replica1)
query:
on one default core: engineeringlogs1_shard1_replica1
{!join from=Maximum_Battery_Charge to=Check_Battery_Charge
fromIndex=engineeringlogs2_shard1_replica1}
Initial_Battery_Charge: "87%"
but stil this issue is coming
find below error:
"error": \{ "metadata": [ "error-class",
"org.apache.solr.common.SolrException", "root-error-class",
"org.apache.solr.common.SolrException" ], "msg": "Cross-core join: no such core
engineeringlogs2_shard1_replica1", "code": 400 } }
Please get me a solution how i can resolve this issue.
was (Author: jyoti609):
Erick:
i guess i am fulfilling all conditonPlease given in points A) exception
handling and B) B) functional enhancement ,
details:
solr version: 4
total nodes: 2
total collection: 2, let say engineeringlogs1, engineeringlogs2
total core in one node: engineeringlogs1_shard1_replica1 and
engineeringlogs2_shard1_replica1
total core in 2nd node: engineeringlogs1_shard2_replica1 and
engineeringlogs2_shard2_replica1
trying to join over over one node between two core,
(engineeringlogs1_shard1_replica1 and engineeringlogs2_shard1_replica1)
query:
on one default core: engineeringlogs1_shard1_replica1
{!join from=Maximum_Battery_Charge to=Check_Battery_Charge
fromIndex=engineeringlogs2_shard1_replica1}Initial_Battery_Charge: "87%"
but stil this issue is coming
find below error:
"error": \{ "metadata": [ "error-class",
"org.apache.solr.common.SolrException", "root-error-class",
"org.apache.solr.common.SolrException" ], "msg": "Cross-core join: no such core
engineeringlogs2_shard1_replica1", "code": 400 } }
Please get me a solution how i can resolve this issue.
> Allow join query over 2 sharded collections: enhance functionality and
> exception handling
> -
>
> Key: SOLR-8297
> URL: https://issues.apache.org/jira/browse/SOLR-8297
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
>Affects Versions: 5.3
>Reporter: Paul Blanchaert
>Priority: Major
> Attachments: SOLR-8297.patch, SOLR-8297_Latest.patch
>
>
> h2. Proposal
> h3. General Idea
> Approach [~shikhasomani]'s range check algorithm to the most cases
> h3. Join behavior depending on router types of joined collections
> || to\\from ||CompositeId||Implicit||
> ||CompositeId| shard range check, see table below | allow |
> ||Implicit| allow | shard to shard |
> h3. CompositeId to CompositeId join behaviour for certain number of shards
>
> || to\\from ||single||>1||
> ||single| allow (as is) | allow (range check) |
> ||>1| allow (as is) | per shard range check |
> h3. Rules from the tables above
> * joining from/to CompositeId and Implicit is blindly allowed, it pick ups
> any collocated replica, because users who do that probably understand what
> they do.
> * when both sides are Implicit let's join shards by name. ie if request hits
> collectionTO_shardY_replica2 at a node, the collocated
> collectionFROM_shardY_replica* is expected.
> * when both sides are CompositeId
> ** from single shard to single shard - nobrainer, just needs collocated
> replica;
> ** from multiple shards to single shard - all "from" shards (any it's
> replicas) are picked for joining
> ** from single shard to multiple shards - existing SOLR-4905 functionality
> ** from multiple to multiple - generic range check algorithm
> ### check that fromField and toField are router.keys in these collections
> ### take shard range for the current "to" collection replica (keep in mind
> that request is distributed across "to" collection shards)
> ### enumerate "from" collection shrads, find their subset which covers "to"
> shard range (this allows to handle any number of shards at both sides)
> ### pickup collocated replicas of these "from" shard subset
> h3. Caveat
> this is quite sensitive to shard allocation (and/or replica placement) ie
> failed "from" replica cannot be collocated with the required "to" shard.
> h2. Initial Description
> Enhancement based on
[jira] [Commented] (SOLR-8297) Allow join query over 2 sharded collections: enhance functionality and exception handling
[
https://issues.apache.org/jira/browse/SOLR-8297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563802#comment-16563802
]
jyoti Tiwari commented on SOLR-8297:
Erick:
i guess i am fulfilling all conditonPlease given in points A) exception
handling and B) B) functional enhancement ,
details:
solr version: 4
total nodes: 2
total collection: 2, let say engineeringlogs1, engineeringlogs2
total core in one node: engineeringlogs1_shard1_replica1 and
engineeringlogs2_shard1_replica1
total core in 2nd node: engineeringlogs1_shard2_replica1 and
engineeringlogs2_shard2_replica1
trying to join over over one node between two core,
(engineeringlogs1_shard1_replica1 and engineeringlogs2_shard1_replica1)
query:
on one default core: engineeringlogs1_shard1_replica1
{!join from=Maximum_Battery_Charge to=Check_Battery_Charge
fromIndex=engineeringlogs2_shard1_replica1}Initial_Battery_Charge: "87%"
but stil this issue is coming
find below error:
"error": \{ "metadata": [ "error-class",
"org.apache.solr.common.SolrException", "root-error-class",
"org.apache.solr.common.SolrException" ], "msg": "Cross-core join: no such core
engineeringlogs2_shard1_replica1", "code": 400 } }
Please get me a solution how i can resolve this issue.
> Allow join query over 2 sharded collections: enhance functionality and
> exception handling
> -
>
> Key: SOLR-8297
> URL: https://issues.apache.org/jira/browse/SOLR-8297
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
>Affects Versions: 5.3
>Reporter: Paul Blanchaert
>Priority: Major
> Attachments: SOLR-8297.patch, SOLR-8297_Latest.patch
>
>
> h2. Proposal
> h3. General Idea
> Approach [~shikhasomani]'s range check algorithm to the most cases
> h3. Join behavior depending on router types of joined collections
> || to\\from ||CompositeId||Implicit||
> ||CompositeId| shard range check, see table below | allow |
> ||Implicit| allow | shard to shard |
> h3. CompositeId to CompositeId join behaviour for certain number of shards
>
> || to\\from ||single||>1||
> ||single| allow (as is) | allow (range check) |
> ||>1| allow (as is) | per shard range check |
> h3. Rules from the tables above
> * joining from/to CompositeId and Implicit is blindly allowed, it pick ups
> any collocated replica, because users who do that probably understand what
> they do.
> * when both sides are Implicit let's join shards by name. ie if request hits
> collectionTO_shardY_replica2 at a node, the collocated
> collectionFROM_shardY_replica* is expected.
> * when both sides are CompositeId
> ** from single shard to single shard - nobrainer, just needs collocated
> replica;
> ** from multiple shards to single shard - all "from" shards (any it's
> replicas) are picked for joining
> ** from single shard to multiple shards - existing SOLR-4905 functionality
> ** from multiple to multiple - generic range check algorithm
> ### check that fromField and toField are router.keys in these collections
> ### take shard range for the current "to" collection replica (keep in mind
> that request is distributed across "to" collection shards)
> ### enumerate "from" collection shrads, find their subset which covers "to"
> shard range (this allows to handle any number of shards at both sides)
> ### pickup collocated replicas of these "from" shard subset
> h3. Caveat
> this is quite sensitive to shard allocation (and/or replica placement) ie
> failed "from" replica cannot be collocated with the required "to" shard.
> h2. Initial Description
> Enhancement based on SOLR-4905. New Jira issue raised as suggested by Mikhail
> Khludnev.
> A) exception handling:
> The exception "SolrCloud join: multiple shards not yet supported" thrown in
> the function findLocalReplicaForFromIndex of JoinQParserPlugin is not
> triggered correctly: In my use-case, I've a join on a facet.query and when my
> results are only found in 1 shard and the facet.query with the join is
> querying the last replica of the last slice, then the exception is not thrown.
> I believe it's better to verify the nr of slices when we want to verify the
> "multiple shards not yet supported" exception (so exception is thrown when
> zkController.getClusterState().getSlices(fromIndex).size()>1).
> B) functional enhancement:
> I would expect that there is no problem to perform a cross-core join over
> sharded collections when the following conditions are met:
> 1) both collections are sharded with the same replicationFactor and numShards
> 2) router.field of the collections is set to the same "key-field" (collection
> of "fromindex" has router.field = "from" field and collection joined to has
> router.field = "to" field)
> The router.field setup ensures that documents
[jira] [Comment Edited] (SOLR-8297) Allow join query over 2 sharded collections: enhance functionality and exception handling
[
https://issues.apache.org/jira/browse/SOLR-8297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563522#comment-16563522
]
jyoti Tiwari edited comment on SOLR-8297 at 7/31/18 11:52 AM:
--
Hi Shikha, i need your help on one issue i am facing currently on solr4 join
query.
i am trying to make cross join query for distributed collection on cloud i.e
engineeringlogs_shard1_replica1 (core1)and
engineeringlogs2_shard1_replica1(core2) for one node, but after join querying
on collection engineeringlogs2 with engineeringlogs:
{!join from=Maximum_Battery_Charge to=Maximum_Battery_Charge
fromIndex=engineeringlogs_shard1_replica1}
Time_Diff_Start_End_BC:"1281"
it is giving error: Cross-core join: no such core
engineeringlogs_shard1_replica1] with root cause
please help me on this issue, whether i can make this cross join query on
single node on solr4.x or i need to upgrade solr version or i am making wrong
solr query.
Please help
was (Author: jyoti609):
Hi Shikha, i need your help on one issue i am facing currently on solr4 join
query.
i am trying to make cross join query for distributed collection on cloud i.e
engineeringlogs_shard1_replica1 and engineeringlogs2_shard1_replica1 for one
node, but after join querying: {!join from=Maximum_Battery_Charge
to=Maximum_Battery_Charge
fromIndex=engineeringlogs_shard1_replica1}Time_Diff_Start_End_BC:"1281"
it is giving error: Cross-core join: no such core
engineeringlogs_shard1_replica1] with root cause
please help me on this issue, whether i can make this cross join query on
single node on solr4.x or i need to upgrade solr version or i am making wrong
solr query.
Please help
> Allow join query over 2 sharded collections: enhance functionality and
> exception handling
> -
>
> Key: SOLR-8297
> URL: https://issues.apache.org/jira/browse/SOLR-8297
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
>Affects Versions: 5.3
>Reporter: Paul Blanchaert
>Priority: Major
> Attachments: SOLR-8297.patch, SOLR-8297_Latest.patch
>
>
> h2. Proposal
> h3. General Idea
> Approach [~shikhasomani]'s range check algorithm to the most cases
> h3. Join behavior depending on router types of joined collections
> || to\\from ||CompositeId||Implicit||
> ||CompositeId| shard range check, see table below | allow |
> ||Implicit| allow | shard to shard |
> h3. CompositeId to CompositeId join behaviour for certain number of shards
>
> || to\\from ||single||>1||
> ||single| allow (as is) | allow (range check) |
> ||>1| allow (as is) | per shard range check |
> h3. Rules from the tables above
> * joining from/to CompositeId and Implicit is blindly allowed, it pick ups
> any collocated replica, because users who do that probably understand what
> they do.
> * when both sides are Implicit let's join shards by name. ie if request hits
> collectionTO_shardY_replica2 at a node, the collocated
> collectionFROM_shardY_replica* is expected.
> * when both sides are CompositeId
> ** from single shard to single shard - nobrainer, just needs collocated
> replica;
> ** from multiple shards to single shard - all "from" shards (any it's
> replicas) are picked for joining
> ** from single shard to multiple shards - existing SOLR-4905 functionality
> ** from multiple to multiple - generic range check algorithm
> ### check that fromField and toField are router.keys in these collections
> ### take shard range for the current "to" collection replica (keep in mind
> that request is distributed across "to" collection shards)
> ### enumerate "from" collection shrads, find their subset which covers "to"
> shard range (this allows to handle any number of shards at both sides)
> ### pickup collocated replicas of these "from" shard subset
> h3. Caveat
> this is quite sensitive to shard allocation (and/or replica placement) ie
> failed "from" replica cannot be collocated with the required "to" shard.
> h2. Initial Description
> Enhancement based on SOLR-4905. New Jira issue raised as suggested by Mikhail
> Khludnev.
> A) exception handling:
> The exception "SolrCloud join: multiple shards not yet supported" thrown in
> the function findLocalReplicaForFromIndex of JoinQParserPlugin is not
> triggered correctly: In my use-case, I've a join on a facet.query and when my
> results are only found in 1 shard and the facet.query with the join is
> querying the last replica of the last slice, then the exception is not thrown.
> I believe it's better to verify the nr of slices when we want to verify the
> "multiple shards not yet supported" exception (so exception is thrown when
> zkController.getClusterState().getSlices(fromIndex).size()>1).
> B) functional enhancement:
> I would expect that there
[jira] [Commented] (SOLR-8297) Allow join query over 2 sharded collections: enhance functionality and exception handling
[
https://issues.apache.org/jira/browse/SOLR-8297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563522#comment-16563522
]
jyoti Tiwari commented on SOLR-8297:
Hi Shikha, i need your help on one issue i am facing currently on solr4 join
query.
i am trying to make cross join query for distributed collection on cloud i.e
engineeringlogs_shard1_replica1 and engineeringlogs2_shard1_replica1 for one
node, but after join querying: {!join from=Maximum_Battery_Charge
to=Maximum_Battery_Charge
fromIndex=engineeringlogs_shard1_replica1}Time_Diff_Start_End_BC:"1281"
it is giving error: Cross-core join: no such core
engineeringlogs_shard1_replica1] with root cause
please help me on this issue, whether i can make this cross join query on
single node on solr4.x or i need to upgrade solr version or i am making wrong
solr query.
Please help
> Allow join query over 2 sharded collections: enhance functionality and
> exception handling
> -
>
> Key: SOLR-8297
> URL: https://issues.apache.org/jira/browse/SOLR-8297
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
>Affects Versions: 5.3
>Reporter: Paul Blanchaert
>Priority: Major
> Attachments: SOLR-8297.patch, SOLR-8297_Latest.patch
>
>
> h2. Proposal
> h3. General Idea
> Approach [~shikhasomani]'s range check algorithm to the most cases
> h3. Join behavior depending on router types of joined collections
> || to\\from ||CompositeId||Implicit||
> ||CompositeId| shard range check, see table below | allow |
> ||Implicit| allow | shard to shard |
> h3. CompositeId to CompositeId join behaviour for certain number of shards
>
> || to\\from ||single||>1||
> ||single| allow (as is) | allow (range check) |
> ||>1| allow (as is) | per shard range check |
> h3. Rules from the tables above
> * joining from/to CompositeId and Implicit is blindly allowed, it pick ups
> any collocated replica, because users who do that probably understand what
> they do.
> * when both sides are Implicit let's join shards by name. ie if request hits
> collectionTO_shardY_replica2 at a node, the collocated
> collectionFROM_shardY_replica* is expected.
> * when both sides are CompositeId
> ** from single shard to single shard - nobrainer, just needs collocated
> replica;
> ** from multiple shards to single shard - all "from" shards (any it's
> replicas) are picked for joining
> ** from single shard to multiple shards - existing SOLR-4905 functionality
> ** from multiple to multiple - generic range check algorithm
> ### check that fromField and toField are router.keys in these collections
> ### take shard range for the current "to" collection replica (keep in mind
> that request is distributed across "to" collection shards)
> ### enumerate "from" collection shrads, find their subset which covers "to"
> shard range (this allows to handle any number of shards at both sides)
> ### pickup collocated replicas of these "from" shard subset
> h3. Caveat
> this is quite sensitive to shard allocation (and/or replica placement) ie
> failed "from" replica cannot be collocated with the required "to" shard.
> h2. Initial Description
> Enhancement based on SOLR-4905. New Jira issue raised as suggested by Mikhail
> Khludnev.
> A) exception handling:
> The exception "SolrCloud join: multiple shards not yet supported" thrown in
> the function findLocalReplicaForFromIndex of JoinQParserPlugin is not
> triggered correctly: In my use-case, I've a join on a facet.query and when my
> results are only found in 1 shard and the facet.query with the join is
> querying the last replica of the last slice, then the exception is not thrown.
> I believe it's better to verify the nr of slices when we want to verify the
> "multiple shards not yet supported" exception (so exception is thrown when
> zkController.getClusterState().getSlices(fromIndex).size()>1).
> B) functional enhancement:
> I would expect that there is no problem to perform a cross-core join over
> sharded collections when the following conditions are met:
> 1) both collections are sharded with the same replicationFactor and numShards
> 2) router.field of the collections is set to the same "key-field" (collection
> of "fromindex" has router.field = "from" field and collection joined to has
> router.field = "to" field)
> The router.field setup ensures that documents with the same "key-field" are
> routed to the same node.
> So the combination based on the "key-field" should always be available within
> the same node.
> From a user perspective, I believe these assumptions seem to be a "normal"
> use-case in the cross-core join in SolrCloud.
> Hope this helps
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
