Re: Getting Error - Session expired for /collections/sprod/state.json

2016-12-16 Thread Erick Erickson 
Look at your connection timeouts and your ZK timeouts.
This usually means your Solr instances are going
into heavy GC as Yago mentions. You can turn on
GC logging if it's not already then use something like
GCViewer to get a handle on the GC.

You really have two options:
1> if it is GC, tune your instances to avoid that if posisble.
This is "more art than science".

2> lengthen timeouts, there are a series of them for client
connections, Solr<->Solr connections and ZK<->Solr
connections

Best,
Erick

On Fri, Dec 16, 2016 at 2:07 AM, Yago Riveiro  wrote:
> Do some gc profiling to get some information about. It's possible you have 
> configure a small heap and you are running in gc stop the world issues.
>
> Normally zookeeper erros are bounded to gc and network latency issues
>
> --
>
> /Yago Riveiro
>
> On 16 Dec 2016, 09:49 +, Piyush Kunal , wrote:
>> Looks like an issue with 6.x version then.
>> But this seems too basic. Not sure if community would not have caught this
>> till now.
>>
>> On Fri, Dec 16, 2016 at 2:55 PM, Yago Riveiro > wrote:
>>
>> > I had some of this error in my logs too on 6.3.0
>> >
>> > My cluster also index like 20K docs/sec I don't know why.
>> >
>> > --
>> >
>> > /Yago Riveiro
>> >
>> > On 16 Dec 2016, 08:39 +, Piyush Kunal ,
>> > wrote:
>> > > Anyone has noticed such issue before?
>> > >
>> > > On Thu, Dec 15, 2016 at 4:36 PM, Piyush Kunal > > > wrote:
>> > >
>> > > > This is happening when heavy indexing like 100/second is going on.
>> > > >
>> > > > On Thu, Dec 15, 2016 at 4:33 PM, Piyush Kunal > > > > wrote:
>> > > >
>> > > > > - We have solr6.1.0 cluster running on production with 1 shard and 5
>> > > > > replicas.
>> > > > > - Zookeeper quorum on 3 nodes.
>> > > > > - Using a chroot in zookeeper to segregate the configs from other
>> > > > > collections.
>> > > > > - Using solrj5.1.0 as our client to query solr.
>> > > > >
>> > > > >
>> > > > >
>> > > > > Usually things work fine but on and off we witness this exception
>> > coming
>> > > > > up:
>> > > > > =
>> > > > > org.apache.solr.common.SolrException: Could not load collection from
>> > > > > ZK:sprod
>> > > > > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
>> > > > > (ZkStateReader.java:815)
>> > > > > at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
>> > > > > er.java:477)
>> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
>> > > > > ection(CloudSolrClient.java:1174)
>> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
>> > > > > hRetryOnStaleState(CloudSolrClient.java:807)
>> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
>> > > > > oudSolrClient.java:782)
>> > > > > --
>> > > > > Caused by: org.apache.zookeeper.KeeperException$
>> > SessionExpiredException:
>> > > > > KeeperErrorCode = Session expired for /collections/sprod/state.json
>> > > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
>> > > > > java:127)
>> > > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
>> > > > > java:51)
>> > > > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>> > > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
>> > > > > ient.java:311)
>> > > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
>> > > > > ient.java:308)
>> > > > > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
>> > > > > CmdExecutor.java:61)
>> > > > > at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
>> > > > > t.java:308)
>> > > > > --
>> > > > > org.apache.solr.common.SolrException: Could not load collection from
>> > > > > ZK:sprod
>> > > > > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
>> > > > > (ZkStateReader.java:815)
>> > > > > at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
>> > > > > er.java:477)
>> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
>> > > > > ection(CloudSolrClient.java:1174)
>> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
>> > > > > hRetryOnStaleState(CloudSolrClient.java:807)
>> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
>> > > > > oudSolrClient.java:782)
>> > > > > --
>> > > > > Caused by: org.apache.zookeeper.KeeperException$
>> > SessionExpiredException:
>> > > > > KeeperErrorCode = Session expired for /collections/sprod/state.json
>> > > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
>> > > > > java:127)
>> > > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
>> > > > > java:51)
>> > > > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>> > > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
>> > > > >

Re: Getting Error - Session expired for /collections/sprod/state.json

2016-12-16 Thread Yago Riveiro 
Do some gc profiling to get some information about. It's possible you have 
configure a small heap and you are running in gc stop the world issues.

Normally zookeeper erros are bounded to gc and network latency issues

--

/Yago Riveiro

On 16 Dec 2016, 09:49 +, Piyush Kunal , wrote:
> Looks like an issue with 6.x version then.
> But this seems too basic. Not sure if community would not have caught this
> till now.
>
> On Fri, Dec 16, 2016 at 2:55 PM, Yago Riveiro  wrote:
>
> > I had some of this error in my logs too on 6.3.0
> >
> > My cluster also index like 20K docs/sec I don't know why.
> >
> > --
> >
> > /Yago Riveiro
> >
> > On 16 Dec 2016, 08:39 +, Piyush Kunal ,
> > wrote:
> > > Anyone has noticed such issue before?
> > >
> > > On Thu, Dec 15, 2016 at 4:36 PM, Piyush Kunal  > > wrote:
> > >
> > > > This is happening when heavy indexing like 100/second is going on.
> > > >
> > > > On Thu, Dec 15, 2016 at 4:33 PM, Piyush Kunal  > > > wrote:
> > > >
> > > > > - We have solr6.1.0 cluster running on production with 1 shard and 5
> > > > > replicas.
> > > > > - Zookeeper quorum on 3 nodes.
> > > > > - Using a chroot in zookeeper to segregate the configs from other
> > > > > collections.
> > > > > - Using solrj5.1.0 as our client to query solr.
> > > > >
> > > > >
> > > > >
> > > > > Usually things work fine but on and off we witness this exception
> > coming
> > > > > up:
> > > > > =
> > > > > org.apache.solr.common.SolrException: Could not load collection from
> > > > > ZK:sprod
> > > > > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
> > > > > (ZkStateReader.java:815)
> > > > > at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
> > > > > er.java:477)
> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
> > > > > ection(CloudSolrClient.java:1174)
> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
> > > > > hRetryOnStaleState(CloudSolrClient.java:807)
> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
> > > > > oudSolrClient.java:782)
> > > > > --
> > > > > Caused by: org.apache.zookeeper.KeeperException$
> > SessionExpiredException:
> > > > > KeeperErrorCode = Session expired for /collections/sprod/state.json
> > > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > > > java:127)
> > > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > > > java:51)
> > > > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> > > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > > > ient.java:311)
> > > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > > > ient.java:308)
> > > > > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
> > > > > CmdExecutor.java:61)
> > > > > at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
> > > > > t.java:308)
> > > > > --
> > > > > org.apache.solr.common.SolrException: Could not load collection from
> > > > > ZK:sprod
> > > > > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
> > > > > (ZkStateReader.java:815)
> > > > > at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
> > > > > er.java:477)
> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
> > > > > ection(CloudSolrClient.java:1174)
> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
> > > > > hRetryOnStaleState(CloudSolrClient.java:807)
> > > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
> > > > > oudSolrClient.java:782)
> > > > > --
> > > > > Caused by: org.apache.zookeeper.KeeperException$
> > SessionExpiredException:
> > > > > KeeperErrorCode = Session expired for /collections/sprod/state.json
> > > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > > > java:127)
> > > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > > > java:51)
> > > > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> > > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > > > ient.java:311)
> > > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > > > ient.java:308)
> > > > > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
> > > > > CmdExecutor.java:61)
> > > > > at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
> > > > > t.java:308)
> > > > > =
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > This is our zoo.cfg:
> > > > > ==
> > > > > tickTime=2000
> > > > > dataDir=/var/lib/zookeeper
> > > > > clientPort=2181
> > > > > initLimit=5
> > > > > syncLimit=2
> > > > > server.1=192.168.70.27:2888:3888
> > > > >

Re: Getting Error - Session expired for /collections/sprod/state.json

2016-12-16 Thread Piyush Kunal 
Looks like an issue with 6.x version then.
But this seems too basic. Not sure if community would not have caught this
till now.

On Fri, Dec 16, 2016 at 2:55 PM, Yago Riveiro 
wrote:

> I had some of this error in my logs too on 6.3.0
>
> My cluster also index like 20K docs/sec I don't know why.
>
> --
>
> /Yago Riveiro
>
> On 16 Dec 2016, 08:39 +, Piyush Kunal ,
> wrote:
> > Anyone has noticed such issue before?
> >
> > On Thu, Dec 15, 2016 at 4:36 PM, Piyush Kunal  > wrote:
> >
> > > This is happening when heavy indexing like 100/second is going on.
> > >
> > > On Thu, Dec 15, 2016 at 4:33 PM, Piyush Kunal  > > wrote:
> > >
> > > > - We have solr6.1.0 cluster running on production with 1 shard and 5
> > > > replicas.
> > > > - Zookeeper quorum on 3 nodes.
> > > > - Using a chroot in zookeeper to segregate the configs from other
> > > > collections.
> > > > - Using solrj5.1.0 as our client to query solr.
> > > >
> > > >
> > > >
> > > > Usually things work fine but on and off we witness this exception
> coming
> > > > up:
> > > > =
> > > > org.apache.solr.common.SolrException: Could not load collection from
> > > > ZK:sprod
> > > > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
> > > > (ZkStateReader.java:815)
> > > > at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
> > > > er.java:477)
> > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
> > > > ection(CloudSolrClient.java:1174)
> > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
> > > > hRetryOnStaleState(CloudSolrClient.java:807)
> > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
> > > > oudSolrClient.java:782)
> > > > --
> > > > Caused by: org.apache.zookeeper.KeeperException$
> SessionExpiredException:
> > > > KeeperErrorCode = Session expired for /collections/sprod/state.json
> > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > > java:127)
> > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > > java:51)
> > > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > > ient.java:311)
> > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > > ient.java:308)
> > > > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
> > > > CmdExecutor.java:61)
> > > > at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
> > > > t.java:308)
> > > > --
> > > > org.apache.solr.common.SolrException: Could not load collection from
> > > > ZK:sprod
> > > > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
> > > > (ZkStateReader.java:815)
> > > > at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
> > > > er.java:477)
> > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
> > > > ection(CloudSolrClient.java:1174)
> > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
> > > > hRetryOnStaleState(CloudSolrClient.java:807)
> > > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
> > > > oudSolrClient.java:782)
> > > > --
> > > > Caused by: org.apache.zookeeper.KeeperException$
> SessionExpiredException:
> > > > KeeperErrorCode = Session expired for /collections/sprod/state.json
> > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > > java:127)
> > > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > > java:51)
> > > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > > ient.java:311)
> > > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > > ient.java:308)
> > > > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
> > > > CmdExecutor.java:61)
> > > > at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
> > > > t.java:308)
> > > > =
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > This is our zoo.cfg:
> > > > ==
> > > > tickTime=2000
> > > > dataDir=/var/lib/zookeeper
> > > > clientPort=2181
> > > > initLimit=5
> > > > syncLimit=2
> > > > server.1=192.168.70.27:2888:3888
> > > > server.2=192.168.70.64:2889:3889
> > > > server.3=192.168.70.26:2889:3889
> > > > maxClientCnxns=300
> > > > maxSessionTimeout=9
> > > > ===
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > This is our solr.xml on server side
> > > > ===
> > > >
> > > >  > > >
> > > >  > > >
> > > > ${host:} > > > ${jetty.port:8983} > > > ${hostContext:solr} > > >
> > > > ${genericCoreNodeNames:true} > > >
> > > > ${zkClientTimeout:3} > > > ${distribUpdateSoTimeout:
> 60}

Re: Getting Error - Session expired for /collections/sprod/state.json

2016-12-16 Thread Yago Riveiro 
I had some of this error in my logs too on 6.3.0

My cluster also index like 20K docs/sec I don't know why.

--

/Yago Riveiro

On 16 Dec 2016, 08:39 +, Piyush Kunal , wrote:
> Anyone has noticed such issue before?
>
> On Thu, Dec 15, 2016 at 4:36 PM, Piyush Kunal  wrote:
>
> > This is happening when heavy indexing like 100/second is going on.
> >
> > On Thu, Dec 15, 2016 at 4:33 PM, Piyush Kunal  > wrote:
> >
> > > - We have solr6.1.0 cluster running on production with 1 shard and 5
> > > replicas.
> > > - Zookeeper quorum on 3 nodes.
> > > - Using a chroot in zookeeper to segregate the configs from other
> > > collections.
> > > - Using solrj5.1.0 as our client to query solr.
> > >
> > >
> > >
> > > Usually things work fine but on and off we witness this exception coming
> > > up:
> > > =
> > > org.apache.solr.common.SolrException: Could not load collection from
> > > ZK:sprod
> > > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
> > > (ZkStateReader.java:815)
> > > at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
> > > er.java:477)
> > > at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
> > > ection(CloudSolrClient.java:1174)
> > > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
> > > hRetryOnStaleState(CloudSolrClient.java:807)
> > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
> > > oudSolrClient.java:782)
> > > --
> > > Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired for /collections/sprod/state.json
> > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > java:127)
> > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > java:51)
> > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > ient.java:311)
> > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > ient.java:308)
> > > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
> > > CmdExecutor.java:61)
> > > at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
> > > t.java:308)
> > > --
> > > org.apache.solr.common.SolrException: Could not load collection from
> > > ZK:sprod
> > > at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
> > > (ZkStateReader.java:815)
> > > at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
> > > er.java:477)
> > > at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
> > > ection(CloudSolrClient.java:1174)
> > > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
> > > hRetryOnStaleState(CloudSolrClient.java:807)
> > > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
> > > oudSolrClient.java:782)
> > > --
> > > Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
> > > KeeperErrorCode = Session expired for /collections/sprod/state.json
> > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > java:127)
> > > at org.apache.zookeeper.KeeperException.create(KeeperException.
> > > java:51)
> > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > ient.java:311)
> > > at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
> > > ient.java:308)
> > > at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
> > > CmdExecutor.java:61)
> > > at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
> > > t.java:308)
> > > =
> > >
> > >
> > >
> > >
> > >
> > > This is our zoo.cfg:
> > > ==
> > > tickTime=2000
> > > dataDir=/var/lib/zookeeper
> > > clientPort=2181
> > > initLimit=5
> > > syncLimit=2
> > > server.1=192.168.70.27:2888:3888
> > > server.2=192.168.70.64:2889:3889
> > > server.3=192.168.70.26:2889:3889
> > > maxClientCnxns=300
> > > maxSessionTimeout=9
> > > ===
> > >
> > >
> > >
> > >
> > >
> > > This is our solr.xml on server side
> > > ===
> > >
> > >  > >
> > >  > >
> > > ${host:} > > ${jetty.port:8983} > > ${hostContext:solr} > >
> > > ${genericCoreNodeNames:true} > >
> > > ${zkClientTimeout:3} > > ${distribUpdateSoTimeout:60} > >  > > name="distribUpdateConnTimeout">${distribUpdateConnTimeout:6} > >  > > name="zkCredentialsProvider">${zkCredentialsProvider:org.apache.solr.common.cloud.DefaultZkCredentialsProvider} > >  > > name="zkACLProvider">${zkACLProvider:org.apache.solr.common.cloud.DefaultZkACLProvider} > >
> > >  > >
> > >  > > class="HttpShardHandlerFactory"
> > > ${socketTimeout:60} > > ${connTimeout:6} > >  > >  > >
> > > ===
> >

Re: Getting Error - Session expired for /collections/sprod/state.json

2016-12-16 Thread Piyush Kunal 
Anyone has noticed such issue before?

On Thu, Dec 15, 2016 at 4:36 PM, Piyush Kunal 
wrote:

> This is happening when heavy indexing like 100/second is going on.
>
> On Thu, Dec 15, 2016 at 4:33 PM, Piyush Kunal 
> wrote:
>
>> - We have solr6.1.0 cluster running on production with 1 shard and 5
>> replicas.
>> - Zookeeper quorum on 3 nodes.
>> - Using a chroot in zookeeper to segregate the configs from other
>> collections.
>> - Using solrj5.1.0 as our client to query solr.
>>
>>
>>
>> Usually things work fine but on and off we witness this exception coming
>> up:
>> =
>> org.apache.solr.common.SolrException: Could not load collection from
>> ZK:sprod
>> at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
>> (ZkStateReader.java:815)
>> at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
>> er.java:477)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
>> ection(CloudSolrClient.java:1174)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
>> hRetryOnStaleState(CloudSolrClient.java:807)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
>> oudSolrClient.java:782)
>> --
>> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired for /collections/sprod/state.json
>> at org.apache.zookeeper.KeeperException.create(KeeperException.
>> java:127)
>> at org.apache.zookeeper.KeeperException.create(KeeperException.
>> java:51)
>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
>> ient.java:311)
>> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
>> ient.java:308)
>> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
>> CmdExecutor.java:61)
>> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
>> t.java:308)
>> --
>> org.apache.solr.common.SolrException: Could not load collection from
>> ZK:sprod
>> at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive
>> (ZkStateReader.java:815)
>> at org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateRead
>> er.java:477)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocColl
>> ection(CloudSolrClient.java:1174)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
>> hRetryOnStaleState(CloudSolrClient.java:807)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
>> oudSolrClient.java:782)
>> --
>> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
>> KeeperErrorCode = Session expired for /collections/sprod/state.json
>> at org.apache.zookeeper.KeeperException.create(KeeperException.
>> java:127)
>> at org.apache.zookeeper.KeeperException.create(KeeperException.
>> java:51)
>> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
>> ient.java:311)
>> at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkCl
>> ient.java:308)
>> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(Zk
>> CmdExecutor.java:61)
>> at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClien
>> t.java:308)
>> =
>>
>>
>>
>>
>>
>> This is our zoo.cfg:
>> ==
>> tickTime=2000
>> dataDir=/var/lib/zookeeper
>> clientPort=2181
>> initLimit=5
>> syncLimit=2
>> server.1=192.168.70.27:2888:3888
>> server.2=192.168.70.64:2889:3889
>> server.3=192.168.70.26:2889:3889
>> maxClientCnxns=300
>> maxSessionTimeout=9
>> ===
>>
>>
>>
>>
>>
>> This is our solr.xml on server side
>> ===
>>
>> 
>>
>>   
>>
>> ${host:}
>> ${jetty.port:8983}
>> ${hostContext:solr}
>>
>> ${genericCoreNodeNames:true}
>>
>> ${zkClientTimeout:3}
>> ${distribUpdateSoTimeout:60}
>> > name="distribUpdateConnTimeout">${distribUpdateConnTimeout:6}
>> > name="zkCredentialsProvider">${zkCredentialsProvider:org.apache.solr.common.cloud.DefaultZkCredentialsProvider}
>> > name="zkACLProvider">${zkACLProvider:org.apache.solr.common.cloud.DefaultZkACLProvider}
>>
>>   
>>
>>   > class="HttpShardHandlerFactory">
>> ${socketTimeout:60}
>> ${connTimeout:6}
>>   
>> 
>>
>> ===
>>
>>
>>
>>
>> Any help appreciated.
>>
>> Regards,
>> Piyush
>>
>
>

Re: Getting Error - Session expired for /collections/sprod/state.json

2016-12-15 Thread Piyush Kunal 
This is happening when heavy indexing like 100/second is going on.

On Thu, Dec 15, 2016 at 4:33 PM, Piyush Kunal 
wrote:

> - We have solr6.1.0 cluster running on production with 1 shard and 5
> replicas.
> - Zookeeper quorum on 3 nodes.
> - Using a chroot in zookeeper to segregate the configs from other
> collections.
> - Using solrj5.1.0 as our client to query solr.
>
>
>
> Usually things work fine but on and off we witness this exception coming
> up:
> =
> org.apache.solr.common.SolrException: Could not load collection from
> ZK:sprod
> at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(
> ZkStateReader.java:815)
> at org.apache.solr.common.cloud.ZkStateReader$5.get(
> ZkStateReader.java:477)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(
> CloudSolrClient.java:1174)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.
> requestWithRetryOnStaleState(CloudSolrClient.java:807)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(
> CloudSolrClient.java:782)
> --
> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /collections/sprod/state.json
> at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:127)
> at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:311)
> at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:308)
> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:61)
> at org.apache.solr.common.cloud.SolrZkClient.exists(
> SolrZkClient.java:308)
> --
> org.apache.solr.common.SolrException: Could not load collection from
> ZK:sprod
> at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(
> ZkStateReader.java:815)
> at org.apache.solr.common.cloud.ZkStateReader$5.get(
> ZkStateReader.java:477)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(
> CloudSolrClient.java:1174)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.
> requestWithRetryOnStaleState(CloudSolrClient.java:807)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(
> CloudSolrClient.java:782)
> --
> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /collections/sprod/state.json
> at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:127)
> at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:311)
> at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:308)
> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:61)
> at org.apache.solr.common.cloud.SolrZkClient.exists(
> SolrZkClient.java:308)
> =
>
>
>
>
>
> This is our zoo.cfg:
> ==
> tickTime=2000
> dataDir=/var/lib/zookeeper
> clientPort=2181
> initLimit=5
> syncLimit=2
> server.1=192.168.70.27:2888:3888
> server.2=192.168.70.64:2889:3889
> server.3=192.168.70.26:2889:3889
> maxClientCnxns=300
> maxSessionTimeout=9
> ===
>
>
>
>
>
> This is our solr.xml on server side
> ===
>
> 
>
>   
>
> ${host:}
> ${jetty.port:8983}
> ${hostContext:solr}
>
> ${genericCoreNodeNames:true}
>
> ${zkClientTimeout:3}
> ${distribUpdateSoTimeout:60}
>  name="distribUpdateConnTimeout">${distribUpdateConnTimeout:6}
>  name="zkCredentialsProvider">${zkCredentialsProvider:org.apache.solr.common.cloud.DefaultZkCredentialsProvider}
>  name="zkACLProvider">${zkACLProvider:org.apache.solr.common.cloud.DefaultZkACLProvider}
>
>   
>
>class="HttpShardHandlerFactory">
> ${socketTimeout:60}
> ${connTimeout:6}
>   
> 
>
> ===
>
>
>
>
> Any help appreciated.
>
> Regards,
> Piyush
>

Getting Error - Session expired for /collections/sprod/state.json

2016-12-15 Thread Piyush Kunal 
- We have solr6.1.0 cluster running on production with 1 shard and 5
replicas.
- Zookeeper quorum on 3 nodes.
- Using a chroot in zookeeper to segregate the configs from other
collections.
- Using solrj5.1.0 as our client to query solr.



Usually things work fine but on and off we witness this exception coming up:
=
org.apache.solr.common.SolrException: Could not load collection from
ZK:sprod
at
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:815)
at
org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateReader.java:477)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1174)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:807)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:782)
--
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /collections/sprod/state.json
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
at
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:308)
at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
at
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)
--
org.apache.solr.common.SolrException: Could not load collection from
ZK:sprod
at
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:815)
at
org.apache.solr.common.cloud.ZkStateReader$5.get(ZkStateReader.java:477)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.getDocCollection(CloudSolrClient.java:1174)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:807)
at
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:782)
--
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /collections/sprod/state.json
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
at
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:308)
at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
at
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:308)
=





This is our zoo.cfg:
==
tickTime=2000
dataDir=/var/lib/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.1=192.168.70.27:2888:3888
server.2=192.168.70.64:2889:3889
server.3=192.168.70.26:2889:3889
maxClientCnxns=300
maxSessionTimeout=9
===





This is our solr.xml on server side
===



  

${host:}
${jetty.port:8983}
${hostContext:solr}

${genericCoreNodeNames:true}

${zkClientTimeout:3}
${distribUpdateSoTimeout:60}
${distribUpdateConnTimeout:6}
${zkCredentialsProvider:org.apache.solr.common.cloud.DefaultZkCredentialsProvider}
${zkACLProvider:org.apache.solr.common.cloud.DefaultZkACLProvider}

  

  
${socketTimeout:60}
${connTimeout:6}
  


===




Any help appreciated.

Regards,
Piyush