Thanks Erick. The numFound is millions but I was also trying with rows= 1 Million. I will reduce it to 500K.
I am sorry. It is state.json. I am using Solr 5.5.0 One of the things I am not able to understand is why my ingestion job is complaining about "Cannot talk to ZooKeeper - Updates are disabled." I have a spark streaming job that continuously ingests into Solr. My shards are always up and running. The moment I start a query on SolrCloud it starts running into this exception. However as you said ZK will only update the state of the cluster when the shards go down. Then why my job is trying to contact ZK when the cluster is up and why is the exception about updating ZK? On Fri, Nov 18, 2016 at 5:11 PM, Erick Erickson <erickerick...@gmail.com> wrote: > The clusterstate on Zookeeper shouldn't be changing > very often, only when nodes come and go. > > bq: At that time I am also running queries (that return > millions of docs). > > As in rows=milions? This is an anti-pattern, if that's true > then you're probably network saturated and the like. If > you mean your numFound is millions, then this is unlikely > to be a problem. > > you say "clusterstate.json", which indicates you're on > 4x? This has been changed to make a state.json for > each collection, so either you upgraded sometime and > didn't transform you ZK (there's a command to do that) > or can you upgrade? > > What I'm guessing is that you have too much going on > somehow and you're overloading your system and > getting a timeout. So increasing the timeout > is definitely a possibility, or reducing the ingestion load > as a test. > > Best, > Erick > > On Fri, Nov 18, 2016 at 4:51 PM, Chetas Joshi <chetas.jo...@gmail.com> > wrote: > > Hi, > > > > I have a SolrCloud (on HDFS) of 50 nodes and a ZK quorum of 5 nodes. The > > SolrCloud is having difficulties talking to ZK when I am ingesting data > > into the collections. At that time I am also running queries (that return > > millions of docs). The ingest job is crying with the the following > exception > > > > org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error > > from server at http://xxx/solr/collection1_shard15_replica1: Cannot > talk to > > ZooKeeper - Updates are disabled. > > > > I think this is happening when the ingest job is trying to update the > > clusterstate.json file but the query is reading from that file and thus > has > > some kind of a lock on that file. Are there any factors that will cause > the > > "READ" to acquire lock for a long time? Is my understanding correct? I am > > using the cursor approach using SolrJ to get back results from Solr. > > > > How often is the ZK updated with the latest cluster state and what > > parameter governs that? Should I just increase the ZK client timeout so > > that it retries connecting to the ZK for a longer period of time (right > now > > it is 15 seconds)? > > > > Thanks! >
