solrcloud 4.3.1 - stability and failure scenario questions

[Utkarsh Sengar]

Hello,

I am testing a 3 node solrcloud cluster with 3 shards. 3 zk nodes are
running in a different process in the same machines.

I wanted to know the recommended size of a solrcloud cluster (min zk nodes?)

This is the SolrCloud dump: https://gist.github.com/utkarsh2012/5840455

And, I am not sure if I am hitting this frustrating bug or this is just a
configuration error from my side. When I kill any *one* of the nodes, the
whole cluster stops responding and I get this request when I query any one
of the two alive nodes.

{
  "responseHeader":{
    "status":503,
    "QTime":2,
    "params":{
      "indent":"true",
      "q":"*:*",
      "wt":"json"}},
  "error":{
    "msg":"no servers hosting shard: ",
    "code":503}}



I see this exception:
952399 [qtp516992923-74] ERROR org.apache.solr.core.SolrCore  –
org.apache.solr.common.SolrException: no servers hosting shard:
    at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:149)
    at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:662)


-- 
Thanks,
-Utkarsh