Re: CloudSolrClient getDocCollection
bq. But I would assume it should still be ok. The number of watchers should still not be gigantic. This assumption would need to be rigorously tested before I'd be comfortable. I've spent quite a bit of time with unhappy clients chasing down issues in the field where 1> it takes hours to cold-start the cluster 2> Solr just locks up 3> etc. That said, there have been a series of other improvements that may have invalidated these concerns. Specifically: 1> Overseer operations were drastically sped up (up to 400x). It's possible that some of the caching was compensating for this. 2> The number of znode changes written was reduced. This would reduce the number of watches triggered and the load on ZK. 3> The number of state changes for nodes coming up was reduced, again reducing the number of watches triggered. 4> etc. But that's hand-waving, I'm mostly saying that the caching code was put in place to solve some existing problem, and I'd hate to have those problems re-introduced. Whether the caching code was the correct solution or whether there are better solutions given additional changes is certainly open for debate. Best, Erick On Sun, Feb 10, 2019 at 3:32 AM Hendrik Haddorp wrote: > > I opened now https://issues.apache.org/jira/browse/SOLR-13239 for the > problem I observed. > > Well, who can really be sure about those things. But I would assume it > should still be ok. The number of watchers should still not be gigantic. > I have setups with about 2000 collections each but far less JVMs. ZK > distributes the watches over the all nodes, which should also include > observer nodes. > > Said that an alternative could be to refresh the cache asynchronously to > the call detecting it to be outdated. Wouldn't the worst case be that a > request gets send to a Solr node that has to forward the request to the > correct node? The chance for the cache entry to be wrong after just one > minute is however quite low. So in most cases the request would still be > send to the correct node without having to wait for the cache update and > without potentially blocking other requests. In a performance test we > saw quite a few threads being blocked at this point. > > regards, > Hendrik > > On 09.02.2019 20:40, Erick Erickson wrote: > > Jason's comments are exactly why there _is_ a state.json per > > collection rather than the single clusterstate.json in the original > > implementation. > > > > Hendrik: > > yes, please do open a JIRA for the condition you observed, > > especially if you can point to the suspect code. There have > > been intermittent issues with collection creation in the test > > shells. > > > > About the watchers. > > > > bq. Yes, you would need one watch per state.json and > > thus one watch per collection. That should however not really be a > > problem with ZK. > > > > Consider an installation I have witnessed with 450K replicas scattered > > over 100s of collections and 100s of JVMs. Each JVM may have one > > or more CloudSolrClients. Are you _sure_ ZK can handle that kind > > of watch load? The current architecture allows there to be many fewer > > watches set, partially to deal with this scale. And even at this scale, > > an incoming request to a node that does _not_ host _any_ replica of > > the target collection needs to be able to forward the request, but doesn't > > need to know much else about the target collections. > > > > Best, > > Erick > > > > > > On Fri, Feb 8, 2019 at 5:23 PM Hendrik Haddorp > > wrote: > >> Hi Jason, > >> > >> thanks for your answer. Yes, you would need one watch per state.json and > >> thus one watch per collection. That should however not really be a > >> problem with ZK. I would assume that the Solr server instances need to > >> monitor those nodes to be up to date on the cluster state. Using > >> org.apache.solr.common.cloud.ZkStateReader.registerCollectionStateWatcher > >> you can even add a watch for that using the SolrJ API. At least for the > >> currently watched collections the client should thus actually already > >> have the correct information available. The access to that would likely > >> be a bit ugly though. > >> > >> The CloudSolrClient also allows to set a watch on /collections using > >> org.apache.solr.common.cloud.ZkStateReader.registerCloudCollectionsListener. > >> This is actually another thing I just ran into. As the code has a watch > >> on /collections the listener gets informed about new collections as soon > >> as the "directory" for the collection is being created. If the listener > >> does then straight away try to access the collection info via > >> zkStateReader.getClusterState() the DocCollection can be returned as > >> null as the DocCollection is build on the information stored in the > >> state.json file, which might not exist yet. I'm trying to monitor the > >> Solr cluster state and thus ran into this. Not sure if I should open a > >> Jira for that. > >> > >> regards, > >> Hendrik > >> > >> On 08.02.2019 23:20, Jason
Re: CloudSolrClient getDocCollection
I opened now https://issues.apache.org/jira/browse/SOLR-13239 for the
problem I observed.
Well, who can really be sure about those things. But I would assume it
should still be ok. The number of watchers should still not be gigantic.
I have setups with about 2000 collections each but far less JVMs. ZK
distributes the watches over the all nodes, which should also include
observer nodes.
Said that an alternative could be to refresh the cache asynchronously to
the call detecting it to be outdated. Wouldn't the worst case be that a
request gets send to a Solr node that has to forward the request to the
correct node? The chance for the cache entry to be wrong after just one
minute is however quite low. So in most cases the request would still be
send to the correct node without having to wait for the cache update and
without potentially blocking other requests. In a performance test we
saw quite a few threads being blocked at this point.
regards,
Hendrik
On 09.02.2019 20:40, Erick Erickson wrote:
Jason's comments are exactly why there _is_ a state.json per
collection rather than the single clusterstate.json in the original
implementation.
Hendrik:
yes, please do open a JIRA for the condition you observed,
especially if you can point to the suspect code. There have
been intermittent issues with collection creation in the test
shells.
About the watchers.
bq. Yes, you would need one watch per state.json and
thus one watch per collection. That should however not really be a
problem with ZK.
Consider an installation I have witnessed with 450K replicas scattered
over 100s of collections and 100s of JVMs. Each JVM may have one
or more CloudSolrClients. Are you _sure_ ZK can handle that kind
of watch load? The current architecture allows there to be many fewer
watches set, partially to deal with this scale. And even at this scale,
an incoming request to a node that does _not_ host _any_ replica of
the target collection needs to be able to forward the request, but doesn't
need to know much else about the target collections.
Best,
Erick
On Fri, Feb 8, 2019 at 5:23 PM Hendrik Haddorp wrote:
Hi Jason,
thanks for your answer. Yes, you would need one watch per state.json and
thus one watch per collection. That should however not really be a
problem with ZK. I would assume that the Solr server instances need to
monitor those nodes to be up to date on the cluster state. Using
org.apache.solr.common.cloud.ZkStateReader.registerCollectionStateWatcher
you can even add a watch for that using the SolrJ API. At least for the
currently watched collections the client should thus actually already
have the correct information available. The access to that would likely
be a bit ugly though.
The CloudSolrClient also allows to set a watch on /collections using
org.apache.solr.common.cloud.ZkStateReader.registerCloudCollectionsListener.
This is actually another thing I just ran into. As the code has a watch
on /collections the listener gets informed about new collections as soon
as the "directory" for the collection is being created. If the listener
does then straight away try to access the collection info via
zkStateReader.getClusterState() the DocCollection can be returned as
null as the DocCollection is build on the information stored in the
state.json file, which might not exist yet. I'm trying to monitor the
Solr cluster state and thus ran into this. Not sure if I should open a
Jira for that.
regards,
Hendrik
On 08.02.2019 23:20, Jason Gerlowski wrote:
Hi Henrik,
I'll try to answer, and let others correct me if I stray. I wasn't
around when CloudSolrClient was written, so take this with a grain of
salt:
"Why does the client need that timeout?Wouldn't it make sense to
use a watch?"
You could probably write a CloudSolrClient that uses watch(es) to keep
track of changing collection state. But I suspect you'd need a
watch-per-collection, instead of just a single watch.
Modern versions of Solr store the state for each collection in
individual "state.json" ZK nodes
("/solr/collections//state.json"). To catch changes
to all of these collections, you'd need to watch each of those nodes.
Which wouldn't scale well for users who want lots of collections. I
suspect this was one of the concerns that nudged the author(s) to use
a cache-based approach.
(Even when all collection state was stored in a single ZK node, a
watch-based CloudSolrClient would likely have scaling issues for the
many-collection use case. The client would need to recalculate its
state information for _all_ collections any time that _any_ of the
collections changed, since it has no way to tell which collection was
changed.)
Best,
Jason
On Thu, Feb 7, 2019 at 11:44 AM Hendrik Haddorp wrote:
Hi,
when I perform a query using the CloudSolrClient the code first
retrieves the DocCollection to determine to which instance the query
should be send [1]. getDocCollection [2] does a lookup in a cache, which
has a 60s expiration time [3].
Re: CloudSolrClient getDocCollection
Jason's comments are exactly why there _is_ a state.json per
collection rather than the single clusterstate.json in the original
implementation.
Hendrik:
yes, please do open a JIRA for the condition you observed,
especially if you can point to the suspect code. There have
been intermittent issues with collection creation in the test
shells.
About the watchers.
bq. Yes, you would need one watch per state.json and
thus one watch per collection. That should however not really be a
problem with ZK.
Consider an installation I have witnessed with 450K replicas scattered
over 100s of collections and 100s of JVMs. Each JVM may have one
or more CloudSolrClients. Are you _sure_ ZK can handle that kind
of watch load? The current architecture allows there to be many fewer
watches set, partially to deal with this scale. And even at this scale,
an incoming request to a node that does _not_ host _any_ replica of
the target collection needs to be able to forward the request, but doesn't
need to know much else about the target collections.
Best,
Erick
On Fri, Feb 8, 2019 at 5:23 PM Hendrik Haddorp wrote:
>
> Hi Jason,
>
> thanks for your answer. Yes, you would need one watch per state.json and
> thus one watch per collection. That should however not really be a
> problem with ZK. I would assume that the Solr server instances need to
> monitor those nodes to be up to date on the cluster state. Using
> org.apache.solr.common.cloud.ZkStateReader.registerCollectionStateWatcher
> you can even add a watch for that using the SolrJ API. At least for the
> currently watched collections the client should thus actually already
> have the correct information available. The access to that would likely
> be a bit ugly though.
>
> The CloudSolrClient also allows to set a watch on /collections using
> org.apache.solr.common.cloud.ZkStateReader.registerCloudCollectionsListener.
> This is actually another thing I just ran into. As the code has a watch
> on /collections the listener gets informed about new collections as soon
> as the "directory" for the collection is being created. If the listener
> does then straight away try to access the collection info via
> zkStateReader.getClusterState() the DocCollection can be returned as
> null as the DocCollection is build on the information stored in the
> state.json file, which might not exist yet. I'm trying to monitor the
> Solr cluster state and thus ran into this. Not sure if I should open a
> Jira for that.
>
> regards,
> Hendrik
>
> On 08.02.2019 23:20, Jason Gerlowski wrote:
> > Hi Henrik,
> >
> > I'll try to answer, and let others correct me if I stray. I wasn't
> > around when CloudSolrClient was written, so take this with a grain of
> > salt:
> >
> > "Why does the client need that timeout?Wouldn't it make sense to
> > use a watch?"
> >
> > You could probably write a CloudSolrClient that uses watch(es) to keep
> > track of changing collection state. But I suspect you'd need a
> > watch-per-collection, instead of just a single watch.
> >
> > Modern versions of Solr store the state for each collection in
> > individual "state.json" ZK nodes
> > ("/solr/collections//state.json"). To catch changes
> > to all of these collections, you'd need to watch each of those nodes.
> > Which wouldn't scale well for users who want lots of collections. I
> > suspect this was one of the concerns that nudged the author(s) to use
> > a cache-based approach.
> >
> > (Even when all collection state was stored in a single ZK node, a
> > watch-based CloudSolrClient would likely have scaling issues for the
> > many-collection use case. The client would need to recalculate its
> > state information for _all_ collections any time that _any_ of the
> > collections changed, since it has no way to tell which collection was
> > changed.)
> >
> > Best,
> >
> > Jason
> >
> > On Thu, Feb 7, 2019 at 11:44 AM Hendrik Haddorp
> > wrote:
> >> Hi,
> >>
> >> when I perform a query using the CloudSolrClient the code first
> >> retrieves the DocCollection to determine to which instance the query
> >> should be send [1]. getDocCollection [2] does a lookup in a cache, which
> >> has a 60s expiration time [3]. When a DocCollection has to be reloaded
> >> this is guarded by a lock [4]. Per default there are 3 locks, which can
> >> cause some congestion. The main question though is why does the client
> >> need that timeout? According to this [5] comment the code does not use a
> >> watch. Wouldn't it make sense to use a watch? I thought the big
> >> advantage of the CloudSolrClient is that is knows were to send requests
> >> to, so that no extra hop needs to be done on the server side. Having to
> >> query ZooKeeper though for the current state does however take some of
> >> that advantage.
> >>
> >> regards,
> >> Hendrik
> >>
> >> [1]
> >> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L849
> >> [2]
> >>
Re: CloudSolrClient getDocCollection
Hi Jason,
thanks for your answer. Yes, you would need one watch per state.json and
thus one watch per collection. That should however not really be a
problem with ZK. I would assume that the Solr server instances need to
monitor those nodes to be up to date on the cluster state. Using
org.apache.solr.common.cloud.ZkStateReader.registerCollectionStateWatcher
you can even add a watch for that using the SolrJ API. At least for the
currently watched collections the client should thus actually already
have the correct information available. The access to that would likely
be a bit ugly though.
The CloudSolrClient also allows to set a watch on /collections using
org.apache.solr.common.cloud.ZkStateReader.registerCloudCollectionsListener.
This is actually another thing I just ran into. As the code has a watch
on /collections the listener gets informed about new collections as soon
as the "directory" for the collection is being created. If the listener
does then straight away try to access the collection info via
zkStateReader.getClusterState() the DocCollection can be returned as
null as the DocCollection is build on the information stored in the
state.json file, which might not exist yet. I'm trying to monitor the
Solr cluster state and thus ran into this. Not sure if I should open a
Jira for that.
regards,
Hendrik
On 08.02.2019 23:20, Jason Gerlowski wrote:
Hi Henrik,
I'll try to answer, and let others correct me if I stray. I wasn't
around when CloudSolrClient was written, so take this with a grain of
salt:
"Why does the client need that timeout?Wouldn't it make sense to
use a watch?"
You could probably write a CloudSolrClient that uses watch(es) to keep
track of changing collection state. But I suspect you'd need a
watch-per-collection, instead of just a single watch.
Modern versions of Solr store the state for each collection in
individual "state.json" ZK nodes
("/solr/collections//state.json"). To catch changes
to all of these collections, you'd need to watch each of those nodes.
Which wouldn't scale well for users who want lots of collections. I
suspect this was one of the concerns that nudged the author(s) to use
a cache-based approach.
(Even when all collection state was stored in a single ZK node, a
watch-based CloudSolrClient would likely have scaling issues for the
many-collection use case. The client would need to recalculate its
state information for _all_ collections any time that _any_ of the
collections changed, since it has no way to tell which collection was
changed.)
Best,
Jason
On Thu, Feb 7, 2019 at 11:44 AM Hendrik Haddorp wrote:
Hi,
when I perform a query using the CloudSolrClient the code first
retrieves the DocCollection to determine to which instance the query
should be send [1]. getDocCollection [2] does a lookup in a cache, which
has a 60s expiration time [3]. When a DocCollection has to be reloaded
this is guarded by a lock [4]. Per default there are 3 locks, which can
cause some congestion. The main question though is why does the client
need that timeout? According to this [5] comment the code does not use a
watch. Wouldn't it make sense to use a watch? I thought the big
advantage of the CloudSolrClient is that is knows were to send requests
to, so that no extra hop needs to be done on the server side. Having to
query ZooKeeper though for the current state does however take some of
that advantage.
regards,
Hendrik
[1]
https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L849
[2]
https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L1180
[3]
https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L162
[4]
https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L1200
[5]
https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L821
Re: CloudSolrClient getDocCollection
Hi Henrik,
I'll try to answer, and let others correct me if I stray. I wasn't
around when CloudSolrClient was written, so take this with a grain of
salt:
"Why does the client need that timeout?Wouldn't it make sense to
use a watch?"
You could probably write a CloudSolrClient that uses watch(es) to keep
track of changing collection state. But I suspect you'd need a
watch-per-collection, instead of just a single watch.
Modern versions of Solr store the state for each collection in
individual "state.json" ZK nodes
("/solr/collections//state.json"). To catch changes
to all of these collections, you'd need to watch each of those nodes.
Which wouldn't scale well for users who want lots of collections. I
suspect this was one of the concerns that nudged the author(s) to use
a cache-based approach.
(Even when all collection state was stored in a single ZK node, a
watch-based CloudSolrClient would likely have scaling issues for the
many-collection use case. The client would need to recalculate its
state information for _all_ collections any time that _any_ of the
collections changed, since it has no way to tell which collection was
changed.)
Best,
Jason
On Thu, Feb 7, 2019 at 11:44 AM Hendrik Haddorp wrote:
>
> Hi,
>
> when I perform a query using the CloudSolrClient the code first
> retrieves the DocCollection to determine to which instance the query
> should be send [1]. getDocCollection [2] does a lookup in a cache, which
> has a 60s expiration time [3]. When a DocCollection has to be reloaded
> this is guarded by a lock [4]. Per default there are 3 locks, which can
> cause some congestion. The main question though is why does the client
> need that timeout? According to this [5] comment the code does not use a
> watch. Wouldn't it make sense to use a watch? I thought the big
> advantage of the CloudSolrClient is that is knows were to send requests
> to, so that no extra hop needs to be done on the server side. Having to
> query ZooKeeper though for the current state does however take some of
> that advantage.
>
> regards,
> Hendrik
>
> [1]
> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L849
> [2]
> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L1180
> [3]
> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L162
> [4]
> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L1200
> [5]
> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L821
CloudSolrClient getDocCollection
Hi, when I perform a query using the CloudSolrClient the code first retrieves the DocCollection to determine to which instance the query should be send [1]. getDocCollection [2] does a lookup in a cache, which has a 60s expiration time [3]. When a DocCollection has to be reloaded this is guarded by a lock [4]. Per default there are 3 locks, which can cause some congestion. The main question though is why does the client need that timeout? According to this [5] comment the code does not use a watch. Wouldn't it make sense to use a watch? I thought the big advantage of the CloudSolrClient is that is knows were to send requests to, so that no extra hop needs to be done on the server side. Having to query ZooKeeper though for the current state does however take some of that advantage. regards, Hendrik [1] https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L849 [2] https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L1180 [3] https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L162 [4] https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L1200 [5] https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/CloudSolrClient.java#L821
