[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917861#comment-16917861
]
ASF subversion and git services commented on SOLR-13257:
Commit 7c101fba4a38e8cd1240d2ee057dc0329f52fea1 in lucene-solr's branch
refs/heads/branch_8x from Anshum Gupta
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=7c101fb ]
SOLR-13257: Cleanup code and make the AffinityReplicaTransformer constructors
private (#848) (#849)
SOLR-13257: Cleanup code and make the constructors private as the constructor
is supposed to be called via the static getInstance method.
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Assignee: Anshum Gupta
>Priority: Minor
> Fix For: master (9.0), 8.3
>
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch,
> SOLR-13257.patch
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917862#comment-16917862
]
ASF subversion and git services commented on SOLR-13257:
Commit 7c101fba4a38e8cd1240d2ee057dc0329f52fea1 in lucene-solr's branch
refs/heads/branch_8x from Anshum Gupta
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=7c101fb ]
SOLR-13257: Cleanup code and make the AffinityReplicaTransformer constructors
private (#848) (#849)
SOLR-13257: Cleanup code and make the constructors private as the constructor
is supposed to be called via the static getInstance method.
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Assignee: Anshum Gupta
>Priority: Minor
> Fix For: master (9.0), 8.3
>
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch,
> SOLR-13257.patch
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917249#comment-16917249
]
ASF subversion and git services commented on SOLR-13257:
Commit 0c9ec35f88fc07aa6ecb0fef2deea705aaee39c2 in lucene-solr's branch
refs/heads/master from Anshum Gupta
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=0c9ec35 ]
SOLR-13257: Cleanup code and make the AffinityReplicaTransformer constructors
private (#848)
SOLR-13257: Cleanup code and make the constructors private as the constructor
is supposed to be called via the static getInstance method.
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Assignee: Anshum Gupta
>Priority: Minor
> Fix For: master (9.0), 8.3
>
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch,
> SOLR-13257.patch
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917248#comment-16917248
]
ASF subversion and git services commented on SOLR-13257:
Commit 0c9ec35f88fc07aa6ecb0fef2deea705aaee39c2 in lucene-solr's branch
refs/heads/master from Anshum Gupta
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=0c9ec35 ]
SOLR-13257: Cleanup code and make the AffinityReplicaTransformer constructors
private (#848)
SOLR-13257: Cleanup code and make the constructors private as the constructor
is supposed to be called via the static getInstance method.
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Assignee: Anshum Gupta
>Priority: Minor
> Fix For: master (9.0), 8.3
>
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch,
> SOLR-13257.patch
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912638#comment-16912638
]
Michael Gibney commented on SOLR-13257:
---
Thanks [~tomasflobbe] and [~cpoerschke]!
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Assignee: Tomás Fernández Löbbe
>Priority: Minor
> Fix For: master (9.0), 8.3
>
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch,
> SOLR-13257.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912598#comment-16912598
]
ASF subversion and git services commented on SOLR-13257:
Commit 706059dded5507ea597e2606dc0ee2e0a7165ee7 in lucene-solr's branch
refs/heads/branch_8x from Tomas Eduardo Fernandez Lobbe
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=706059d ]
SOLR-13257: Support deterministic replica routing
Deterministic replica routing can help improve caching and allow a more
consistent paging when sorting by score
This closes #677
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Assignee: Tomás Fernández Löbbe
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch,
> SOLR-13257.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16912591#comment-16912591
]
ASF subversion and git services commented on SOLR-13257:
Commit 8f4103dd4b7d572300aa585b9e25120fa97a25f3 in lucene-solr's branch
refs/heads/master from Tomas Eduardo Fernandez Lobbe
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=8f4103d ]
SOLR-13257: Support deterministic replica routing
Deterministic replica routing can help improve caching and allow a more
consistent paging when sorting by score
This closes #677
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Assignee: Tomás Fernández Löbbe
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch,
> SOLR-13257.patch
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910642#comment-16910642
]
Christine Poerschke commented on SOLR-13257:
+1 i.e. ready to merge from my perspective too, thanks.
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Assignee: Tomás Fernández Löbbe
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch,
> SOLR-13257.patch
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909874#comment-16909874
]
Michael Gibney commented on SOLR-13257:
---
No more changes; looks ready to merge from my perspective. Thanks!
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Assignee: Tomás Fernández Löbbe
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch,
> SOLR-13257.patch
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16909387#comment-16909387
]
Tomás Fernández Löbbe commented on SOLR-13257:
--
Is this ready to merge? or are you doing more changes [~mgibney]?
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Assignee: Tomás Fernández Löbbe
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch,
> SOLR-13257.patch
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900547#comment-16900547
]
Michael Gibney commented on SOLR-13257:
---
Thanks [~tomasflobbe] and [~cpoerschke] for your feedback, and for helping move
this feature along! I just pushed some commits to address some points raised by
[~cpoerschke]: opting for stricter handling of the {{dividend}} param (if
present, it must be an integer), and adding/clarifying documentation and
javadocs.
Most of the changes have to do with added documentation. One functional change
was to add support for the ability to explicitly disable implicit hash-based
routing (by setting the {{hash}} param config to the empty String, in
solr.xml). This idea came up as a consequence of thinking about edge cases
while documenting the behavior of replica routing configuration in solr.xml.
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Assignee: Tomás Fernández Löbbe
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch,
> SOLR-13257.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899135#comment-16899135
]
Christine Poerschke commented on SOLR-13257:
Thanks [~tomasflobbe] for the ping! I left some comments on the pull requests
but have no concerns, it's a very elegant code solution indeed, thank you
[~mgibney] for developing and contributing it.
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Assignee: Tomás Fernández Löbbe
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch,
> SOLR-13257.patch
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899041#comment-16899041
]
Tomás Fernández Löbbe commented on SOLR-13257:
--
Sorry [~mgibney], I've been quite busy these days. Code looks good, I'll merge
soon unless there are any concerns ([~cpoerschke]?)
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Assignee: Tomás Fernández Löbbe
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch,
> SOLR-13257.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882628#comment-16882628
]
Michael Gibney commented on SOLR-13257:
---
Thanks, [~cpoerschke] and [~tomasflobbe]! Responses shortly to PR comments.
[~cpoerschke], I think the examples (and answers) you set out are all
illustrative and accurate (with the exception that the syntax would be
{{shards.preference=replica.base:stable:dividend:fooBar=0}} or
{{shards.preference=replica.base:stable:hash:fooBar=0)}}
A couple of comments on the initial example cases (though again it seems that
the the answers you provide are accurate as far as I understand it):
2.Example: opt for possibility 2: It's a nitpicky semantic point, but the
deterministic preference param doesn't ever really resolve directly to a
particular replica, rather it resolves to a particular rotation of the replica
preferences list. The first of these will in most (all?) cases be chosen to
serve the request, but there's nothing special about the "top" replica _per
se_, and thus nothing different about the _other_ replicas (that would cause
them to be sorted differently).
4. "Does the new shard affinity logic influence the ordering within the end
portion of the list too?"
Yes; each grouping of otherwise equivalent options is sorted and
deterministically rotated according to the specified affinity param.
5. Same as 4; the new affinity logic only affects groupings of options that are
otherwise (according to specified {{shards.preference}} param) considered to be
equivalent; and it affects all such groupings, no matter how many hierarchical
preferences are specified, or how many otherwise-equivalent groups there are.
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch,
> SOLR-13257.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882505#comment-16882505
]
Tomás Fernández Löbbe commented on SOLR-13257:
--
This feature sounds great. I left some comments in the PR (not sure if the PR
has the latest patch or not)
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch,
> SOLR-13257.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882405#comment-16882405
]
Christine Poerschke commented on SOLR-13257:
Thanks [~mgibney] for creating and working on this ticket!
I have some "requirements" or "intended behaviour" type questions or thoughts,
perhaps best posed via examples.
1. The out-of-the-box behaviour (in the absence of {{shards.preference}} or
similar parameters) is, as you say, a random choice.
*
** Example: A collection with three replicas (replicaA, replicaB, replicaC),
all of which are available at the time of searching.
** Expected behaviour: The list of available replicas is randomly shuffled
i.e. there are {{3! = 6}} possibilities overall: {{[ [A, B, C], [A, C, B], [B,
A, C], [B, C, A], [C, A, B], [C, B, A] ]}}
2. The objective of this ticket i.e. the proposed behaviour (in the absence of
{{shards.preference}} or similar parameters) is reduce or remove the
random-ness of the out-of-the-box behaviour.
*
** Example: A collection with three replicas (replicaA, replicaB, replicaC),
all of which are available at the time of searching. The request indicates
(directly via a numeric value parameter or indirectly via a non-numeric value
parameter that can be hashed) the affinity to an element in or a portion of the
list of replicas. Let's assume the affinity expressed translates into replica
{{A}}.
** Expected behaviour:
*** Possibility 1: The preferred replica is {{A}} but beyond the first choice
the random-ness remains for load balancing purposes i.e. there are {{2! = 2}}
possibilities overall: {{[ [A, B, C], [A, C, B] ]}}
*** Possibility 2: The preferred replica is {{A}} and also beyond that first
choice there is no random-ness i.e. the replica list is sorted
deterministically to give just one possibility: {{[A, B, C]}}
3. The existing {{shards.preference}} parameter influences the replica choices
(in the absence of new affinity logic).
(caveat: I presume the below is the existing behaviour of the existing code but
have not recently read the code in detail to fully convince myself that this is
so.)
*
** Example 1: A collection with six replicas spread equally across three
locations. The request indicates a preference for "Europe" location.
** Expected behaviour: The preferred replicas (based on location) are at the
start of the list and the not-preferred replicas are at the end of the list.
The ordering within the start and end portions of the list is random giving
{{2! * 4! = 2*1 * 4*3*2*1 = 2 * 24 = 48}} possibilities overall.
*
** Example 2: A collection with six replicas spread equally across three
locations and the two replicas at each location are of different types e.g.
{{PULL}} and {{TLOG}}. The request indicates a preference primarily for
"Europe" location and secondarily for "PULL" type.
** Expected behaviour: The preferred replicas (based on location) are at the
start of the list and the not-preferred replicas are at the end of the list.
The ordering within the start portion of the list becomes deterministic via the
secondary preference (e.g. {{[ replicaB1(europe,pull), replicaB2(europe,tlog)
]}}) and the ordering within the end portion of the list is narrowed down via
the secondary preference but some random-ness remains giving 4 possibilities
overall:
{code:java}
[ replicaB1(europe,pull), replicaB2(europe,tlog),
replicaA1(america,pull), replicaC1(asia,pull),
replicaA2(america,tlog), replicaC2(asia,tlog) ]
[ replicaB1(europe,pull), replicaB2(europe,tlog),
replicaA1(america,pull), replicaC1(asia,pull),
replicaC2(asia,tlog), replicaA2(america,tlog) ]
[ replicaB1(europe,pull), replicaB2(europe,tlog),
replicaC1(asia,pull), replicaA1(america,pull),
replicaC2(asia,tlog), replicaA2(america,tlog) ]
[ replicaB1(europe,pull), replicaB2(europe,tlog),
replicaC1(asia,pull), replicaA1(america,pull),
replicaA2(america,tlog), replicaC2(asia,tlog) ]
{code}
4. The new shard affinity logic combined with a single {{shards.preference}}
parameter.
*
** Example: A collection with six replicas spread equally across three
locations. The request indicates a preference for "Europe" location.
** Expected behaviour: The preferred replicas (based on location) are at the
start of the list and the not-preferred replicas are at the end of the list.
*** The ordering within the "Europe" start portion of the list becomes
deterministic via the new shard affinity logic.
*** Does the new shard affinity logic influence the ordering within the end
portion of the list too?
5. The new shard affinity logic combined with multiple {{shards.preference}}
parameters.
*
** At its simplest each {{shards.preference}} parameter will have two choices
e.g. "PULL" vs. "TLOG" replica type or "Europe" vs. unknown replica location.
** Assuming two {{shards.preference}} parameters with two choices each the
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881478#comment-16881478
]
David Smiley commented on SOLR-13257:
-
Cool!
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch,
> SOLR-13257.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881072#comment-16881072
]
Lucene/Solr QA commented on SOLR-13257:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 2m 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 1m 54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 1m 54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate ref guide {color} |
{color:green} 1m 54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 31m
25s{color} | {color:green} core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
55s{color} | {color:green} solrj in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-13257 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12973965/SOLR-13257.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns validaterefguide |
| uname | Linux lucene1-us-west 4.4.0-137-generic #163~14.04.1-Ubuntu SMP Mon
Sep 24 17:14:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 0c09481 |
| ant | version: Apache Ant(TM) version 1.9.3 compiled on July 24 2018 |
| Default Java | LTS |
| Test Results |
https://builds.apache.org/job/PreCommit-SOLR-Build/484/testReport/ |
| modules | C: solr solr/core solr/solrj solr/solr-ref-guide U: solr |
| Console output |
https://builds.apache.org/job/PreCommit-SOLR-Build/484/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch,
> SOLR-13257.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880136#comment-16880136
]
Lucene/Solr QA commented on SOLR-13257:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
42s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 2m 10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 1m 57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 1m 57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate ref guide {color} |
{color:green} 1m 57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 44s{color}
| {color:red} core in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 2s{color}
| {color:red} solrj in the patch failed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | solr.TestRandomFaceting |
| | solr.search.similarities.TestNonDefinedSimilarityFactory |
| | solr.search.similarities.TestLMJelinekMercerSimilarityFactory |
| | solr.response.TestRawTransformer |
| | solr.client.solrj.embedded.SolrExampleEmbeddedTest |
| | solr.client.solrj.GetByIdTest |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-13257 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12973864/SOLR-13257.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns validaterefguide |
| uname | Linux lucene1-us-west 4.4.0-137-generic #163~14.04.1-Ubuntu SMP Mon
Sep 24 17:14:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / ac209b6 |
| ant | version: Apache Ant(TM) version 1.9.3 compiled on July 24 2018 |
| Default Java | LTS |
| unit |
https://builds.apache.org/job/PreCommit-SOLR-Build/482/artifact/out/patch-unit-solr_core.txt
|
| unit |
https://builds.apache.org/job/PreCommit-SOLR-Build/482/artifact/out/patch-unit-solr_solrj.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-SOLR-Build/482/testReport/ |
| modules | C: solr/core solr/solrj solr/solr-ref-guide U: solr |
| Console output |
https://builds.apache.org/job/PreCommit-SOLR-Build/482/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Reporter: Michael Gibney
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
>
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879951#comment-16879951
]
Jan Høydahl commented on SOLR-13257:
I extracted a patch file from your PR and attached here to trigger automatic
patch validation. Hopefully there will be a post from Yetus soon with either +1
or -1 result, which will indicate whether more work is needed on the patch. I
also tagged [~tomasflobbe] in the PR as a potential reviewer.
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
>Affects Versions: 7.4, master (9.0)
>Reporter: Michael Gibney
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839653#comment-16839653
]
Michael Gibney commented on SOLR-13257:
---
[PR created|https://github.com/apache/lucene-solr/pull/677]. This ended up
being a bit more involved than I'd anticipated, because I ended up integrating
replica affinity with {{shards.preference}}. This seemed like the most natural
place to add this functionality, and it will allow deterministic routing to be
used in conjunction with other replica routing preferences (as a
fallback/tie-breaker). There's also support for configuring default "base
routing" preference in solrconfig.xml.
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Affects Versions: 7.4, master (9.0)
>Reporter: Michael Gibney
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java
>
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806843#comment-16806843
]
Michael Gibney commented on SOLR-13257:
---
Thanks for the feedback, [~janhoy]! Good point re: {{getReplicaId()}} -- I
think that was "proof-of-concept" code that escaped my notice, and should no
doubt be replaced. I'm happy to submit as a PR, and will work on a test.
{quote}I wonder if this perhaps is such a common use case that we could
consider folding it into the default shardHandler as a configuration option.
{quote}
I had been wondering this as well. I think that would make sense, but I wanted
to get something out there initially that people could use without patching
core classes. I think for a PR I'll try the "config option in default
shardHandler" approach. Will be easy to extract again if necessary.
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Affects Versions: 7.4, master (9.0)
>Reporter: Michael Gibney
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java
>
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806730#comment-16806730
]
Jan Høydahl commented on SOLR-13257:
I wonder if this perhaps is such a common use case that we could consider
folding it into the default shardHandler as a configuration option.
Also, the {{getReplicaId()}} method looks a bit fragile relying on number
encoded in core name. I think it may break if someone hand-chooses core name.
So please try to pull the replica number from somewhere else.
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Affects Versions: 7.4, master (9.0)
>Reporter: Michael Gibney
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java
>
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage
[
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806683#comment-16806683
]
Jan Høydahl commented on SOLR-13257:
This looks promising. Are you able to open a Pull Request and submit this code?
See [https://wiki.apache.org/solr/HowToContribute#Working_with_GitHub] for how.
Also we need a JUnit test case that proves that the new code actually does what
it promises to.
> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Affects Versions: 7.4, master (9.0)
>Reporter: Michael Gibney
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java
>
>
> For each shard in a distributed request, Solr currently routes each request
> randomly via
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
> to a particular replica. In setups with replication factor >1, this normally
> results in a situation where subsequent requests (which one would hope/expect
> to leverage cached results from previous related requests) end up getting
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe
> containing common terms?). The first request initializes the
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and
> there are a sufficient number of shards, subsequent requests will likely be
> routed to at least one replica that _hasn't_ seen the query before. The
> replicas with uninitialized caches become a bottleneck, and from the client's
> perspective, many subsequent requests appear not to benefit from caching at
> all.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
