[jira] [Commented] (SOLR-9014) Audit all usages of ClusterState methods which may make calls to ZK via the lazy collection reference
[ https://issues.apache.org/jira/browse/SOLR-9014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254121#comment-15254121 ] Shalin Shekhar Mangar commented on SOLR-9014: - I found SOLR-9030 while working on this issue. > Audit all usages of ClusterState methods which may make calls to ZK via the > lazy collection reference > - > > Key: SOLR-9014 > URL: https://issues.apache.org/jira/browse/SOLR-9014 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Reporter: Shalin Shekhar Mangar > Fix For: master, 6.1 > > > ClusterState has a bunch of methods such as getSlice and getReplica which > internally call getCollectionOrNull that ends up making a call to ZK via the > lazy collection reference. Many classes use these methods even though a > DocCollection object is available. In such cases, multiple redundant calls to > ZooKeeper can happen if the collection is not watched locally. This is > especially true for Overseer classes which operate on all collections. > We should audit all usages of these methods and replace them with calls to > appropriate DocCollection methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9014) Audit all usages of ClusterState methods which may make calls to ZK via the lazy collection reference
[ https://issues.apache.org/jira/browse/SOLR-9014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251031#comment-15251031 ] Shalin Shekhar Mangar commented on SOLR-9014: - You are right. I was confused. I haven't dug into SOLR-8323 yet but if you say so :-) > Audit all usages of ClusterState methods which may make calls to ZK via the > lazy collection reference > - > > Key: SOLR-9014 > URL: https://issues.apache.org/jira/browse/SOLR-9014 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Reporter: Shalin Shekhar Mangar > Fix For: master, 6.1 > > > ClusterState has a bunch of methods such as getSlice and getReplica which > internally call getCollectionOrNull that ends up making a call to ZK via the > lazy collection reference. Many classes use these methods even though a > DocCollection object is available. In such cases, multiple redundant calls to > ZooKeeper can happen if the collection is not watched locally. This is > especially true for Overseer classes which operate on all collections. > We should audit all usages of these methods and replace them with calls to > appropriate DocCollection methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9014) Audit all usages of ClusterState methods which may make calls to ZK via the lazy collection reference
[ https://issues.apache.org/jira/browse/SOLR-9014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250898#comment-15250898 ] Scott Blum commented on SOLR-9014: -- Hard to get the right context in mind in all these cases, eh? 1. If the overseer had set a watch on the collection it cared about, it would be more efficient to wait and loop. Maybe we could fix this via SOLR-8323? Overseer could set a temporary watch on things it cared about. One thing puzzles me in what you said tho; if the collection is lazy but existent, I think forceUpdateCollection() is basically exactly as efficient as just polling the lazy collection. So going back to forceUpdateCollection() won't make it any better. But in cases where the collection is watching, looping is far more efficient. 2) Agreed, maybe we should expose getCollectionStates() or getCollectionNames(). > Audit all usages of ClusterState methods which may make calls to ZK via the > lazy collection reference > - > > Key: SOLR-9014 > URL: https://issues.apache.org/jira/browse/SOLR-9014 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Reporter: Shalin Shekhar Mangar > Fix For: master, 6.1 > > > ClusterState has a bunch of methods such as getSlice and getReplica which > internally call getCollectionOrNull that ends up making a call to ZK via the > lazy collection reference. Many classes use these methods even though a > DocCollection object is available. In such cases, multiple redundant calls to > ZooKeeper can happen if the collection is not watched locally. This is > especially true for Overseer classes which operate on all collections. > We should audit all usages of these methods and replace them with calls to > appropriate DocCollection methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9014) Audit all usages of ClusterState methods which may make calls to ZK via the lazy collection reference
[ https://issues.apache.org/jira/browse/SOLR-9014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15250642#comment-15250642 ] Shalin Shekhar Mangar commented on SOLR-9014: - This is turning out to be interesting. We have to revisit a few assumptions: # The Overseer has a wait loop to see a certain condition to be true in many places. The earlier assumption was that updateClusterState was expensive and therefore it was better to wait until you see the state. But not that we have lazy collections and a collection specific forceUpdateCollection, the wait look is actually more expensive because it ends up reading the collection state from ZooKeeper -- sometime as frequently as 100ms. # The ClusterState#getCollections was supposed to be lightweight i.e. it just read and returned the names of known collections from local cached state. This was changed in SOLR-6629 to resolve the reference. This means that it ends up going to ZK for each non-watched collection. So API calls like LIST, downnode etc have become way more expensive. It is better to start returning a List from this method instead. > Audit all usages of ClusterState methods which may make calls to ZK via the > lazy collection reference > - > > Key: SOLR-9014 > URL: https://issues.apache.org/jira/browse/SOLR-9014 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Reporter: Shalin Shekhar Mangar > Fix For: master, 6.1 > > > ClusterState has a bunch of methods such as getSlice and getReplica which > internally call getCollectionOrNull that ends up making a call to ZK via the > lazy collection reference. Many classes use these methods even though a > DocCollection object is available. In such cases, multiple redundant calls to > ZooKeeper can happen if the collection is not watched locally. This is > especially true for Overseer classes which operate on all collections. > We should audit all usages of these methods and replace them with calls to > appropriate DocCollection methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
