vthacker opened a new pull request #2012:
URL: https://github.com/apache/lucene-solr/pull/2012


   
   
   # Description
   
   When querying using SolrJ today this is what happens - 
   - CloudSolrClient creates a list of all replicas from the alias ( after 
resolving it ) and then shuffles the list picking one Solr node to query against
   - When Solr receives the request, it only looks at the first collection from 
the alias and tries to find a local core
   - Anytime that isn't the case, it makes a remoteQuery ,  proxying the 
request to machineX which has a local core of the first collection from the 
alias.
   
   # Solution
   
   The solution involves trying to find a local core by looking at all the 
collections in the list and not just the first collection. this is a cheap 
operation and saves the overhead of an extra network hop
   
   # Tests
   
   
   # Performance
   
   I was lucky enough to have a cluster which had exact production 
characteristics but was in dark mode. So we were able to test this change out 
internally and check out it's effectiveness 
   
   After a rolling restart of the cluster we saw the CPU idle percentage 
increase by roughly 10%
   
   
![image](https://user-images.githubusercontent.com/158041/96658582-ba7f6580-12f9-11eb-9c42-a88356d10e36.png)
   
   
   This improved the end to end latency as well. The graph purposely doesn't 
have the Y axis but it's a stack graph that buckets latency ( for example what 
% queries fall bellow 100ms etc ) . So the base green and yellow lines 
increasing means we are able to serve queries faster.
   
   
![image](https://user-images.githubusercontent.com/158041/96658338-172e5080-12f9-11eb-9a74-e7a547972248.png)
   
   I don't expect the latency improvements to be this drastic for everyone. In 
our specific case we had 2 things that really improved with this change
   
   1. We have ~15 collections in an alias. So there was only a 14/15 times the 
query would end up being a remote query for searches
   2. These remoteQueries were going through PKI authentication and not 
BasicAuth ( we use `forwardCredentals=true` ) . In our flame graphs we had seen 
this take a significant percentage of wall clock time ( 30% ) .
   
   
![image](https://user-images.githubusercontent.com/158041/96658881-89ebfb80-12fa-11eb-916d-64171b581c61.png)
   
   When we made the change to use `forwardCredentals=true` we saw this reduce to
   
   
![image](https://user-images.githubusercontent.com/158041/96658948-ad16ab00-12fa-11eb-9612-864db2bf715b.png)
    
   It is when we saw PKIAuth still show up in the flemagraph that we noticed 
`remoteQuery` being the reason
   
   After the rollout of this PR this PKI isn't visible in the flamegraph at all 
🎉 
   
   
![image](https://user-images.githubusercontent.com/158041/96659068-e94a0b80-12fa-11eb-9272-30aa2246d143.png)
   
   # Checklist
   
   Please review the following and check all that apply:
   
   - [ ] I have reviewed the guidelines for [How to 
Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms 
to the standards described there to the best of my ability.
   - [ ] I have created a Jira issue and added the issue ID to my pull request 
title.
   - [ ] I have given Solr maintainers 
[access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork)
 to contribute to my PR branch. (optional but recommended)
   - [ ] I have developed this patch against the `master` branch.
   - [ ] I have run `./gradlew check`.
   - [ ] I have added tests for my changes.
   - [ ] I have added documentation for the [Ref 
Guide](https://github.com/apache/lucene-solr/tree/master/solr/solr-ref-guide) 
(for Solr changes only).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[email protected]



---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to