Sorry Erick, I missed your reply because it went to the mailing list and not to the Jira.
This issue is a little different. What you describe is a completely valid bug but separate from this one. The ClusterState class has some helper methods and we use them in places where a DocCollection is already available. These helper methods end up making another call to ZK which would be totally unnecessary if the calling code had queried the DocCollection object for that information. So this issue is just about removing usages of such methods so that we don't make redundant calls to ZK. On Wed, Apr 27, 2016 at 2:20 AM, Erick Erickson <[email protected]> wrote: > I'm not sure this is an edge case at all. LazyCollectionRef goes to > Zookeeper for any collection that does _not_ have a replica on the > Solr node in question IIUC. There's even a comment in the "get()" > method: > > // TODO: consider limited caching > > So consider someone sending HTTP request to update documents to Solr. > Or even queries. Or fronting the cluster with a load-balancer. Any > request that ends up on a Solr node that does not host at least one > replica for that collection will cause that node to go to ZK to get > the collection state _even if it's just been fetched_. > > I've seen a situation "in the field" where there are over 1,500 > threads on a Solr client going to ZK to get a collection state. I > suspect there's something _else_ going on here that causes the SolrJ > client (this particular client is certainly using SolrJ) to send the > updates to the wrong Solr node or something, but still this is > egregious. We really shouldn't need to pull the state from ZK so often > in that case. > > OTOH, we don't want to cache the collection state (and set a watch) > for any collection ever referenced by a Solr instance or we might as > well go back to the single clusterstate.json.... > > Or I completely misunderstand the situation, wouldn't be the first time. > > On Tue, Apr 26, 2016 at 11:06 AM, Scott Blum (JIRA) <[email protected]> > wrote: > > > > [ > https://issues.apache.org/jira/browse/SOLR-9014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15258576#comment-15258576 > ] > > > > Scott Blum commented on SOLR-9014: > > ---------------------------------- > > > > Correct, the edge case is what we're guarding against here. I would be > completely on board with making collectionStates() cheap, but we would need > to go update all the call sites to guard against that edge case; ie, using > a key returned from collectionStates(), all the call sites need to expect > that getCollection(key) might return null. > > > >> Deprecate and reduce usage of ClusterState methods which may make calls > to ZK via the lazy collection reference > >> > --------------------------------------------------------------------------------------------------------------- > >> > >> Key: SOLR-9014 > >> URL: https://issues.apache.org/jira/browse/SOLR-9014 > >> Project: Solr > >> Issue Type: Improvement > >> Components: SolrCloud > >> Reporter: Shalin Shekhar Mangar > >> Fix For: master, 6.1 > >> > >> Attachments: SOLR-9014.patch > >> > >> > >> ClusterState has a bunch of methods such as getSlice and getReplica > which internally call getCollectionOrNull that ends up making a call to ZK > via the lazy collection reference. Many classes use these methods even > though a DocCollection object is available. In such cases, multiple > redundant calls to ZooKeeper can happen if the collection is not watched > locally. This is especially true for Overseer classes which operate on all > collections. > >> We should audit all usages of these methods and replace them with calls > to appropriate DocCollection methods. > > > > > > > > -- > > This message was sent by Atlassian JIRA > > (v6.3.4#6332) > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- Regards, Shalin Shekhar Mangar.
