Re: No live SolrServers available to handle this request

[Anil]

Thanks Shawn and Elaine,

Elaine,
Yes all the documents of same route key resides on same shard.

Shawn,
I will try to capture the logs.

Thanks.

Regards,
Anil



On 25 March 2016 at 02:57, Elaine Cario  wrote:

> Anil,
>
> I've seen situations where if there was a problem with a specific query,
> and every shard responds with the same error, the actual exception gets
> hidden  by a "No live SolrServers..." exception.  We originally saw this
> with wildcard queries (when every shard reported a "too many expansions..."
> type error, but the exception in the response was "No live SolrServers..."
> error.
>
> You mention that you are using collapse/expand, and that you have shards -
> that could possibly cause some issue, as I think collapse and expand only
> work correctly if the data for any particular collapse value resides on one
> shard.
>
> On Sat, Mar 19, 2016 at 1:04 PM, Shawn Heisey  wrote:
>
> > On 3/18/2016 9:55 PM, Anil wrote:
> > > Thanks for your response.
> > > CDH is a Cloudera (third party) distribution. is there any to get the
> > > notifications copy of it when cluster state changed ? in logs ?
> > >
> > > I can assume that the exception is result of no availability of
> replicas
> > > only. Agree?
> >
> > Yes, I think that Solr believes there are no replicas for at least one
> > shard.  As for why it believes that, I cannot say.
> >
> > If Solr logged every single thing that happened where zookeeper (or even
> > just the clusterstate) is involved, you'd be drowning in logs.  Much
> > more than already happens.  The logfile is already very verbose.
> >
> > Chances are that at least one of your Solr nodes *did* log something
> > related to a problem with that collection before you got the error
> > you're asking about.
> >
> > The "No live SolrServers" error is one that people are seeing quite
> > frequently.  There may be some instances where Solr isn't behaving
> > correctly, but I think when this happens, it usually indicates there's a
> > real problem of some kind.
> >
> > To troubleshoot, we'll need to see any errors or warnings you find in
> > your Solr logfiles from the time before you get an error on a request.
> > You'll need to check the logfile on all Solr nodes.
> >
> > It might be a good idea to also involve Cloudera support, see what they
> > think.
> >
> > Thanks,
> > Shawn
> >
> >
>

Re: No live SolrServers available to handle this request

[Elaine Cario]

Anil,

I've seen situations where if there was a problem with a specific query,
and every shard responds with the same error, the actual exception gets
hidden  by a "No live SolrServers..." exception.  We originally saw this
with wildcard queries (when every shard reported a "too many expansions..."
type error, but the exception in the response was "No live SolrServers..."
error.

You mention that you are using collapse/expand, and that you have shards -
that could possibly cause some issue, as I think collapse and expand only
work correctly if the data for any particular collapse value resides on one
shard.

On Sat, Mar 19, 2016 at 1:04 PM, Shawn Heisey  wrote:

> On 3/18/2016 9:55 PM, Anil wrote:
> > Thanks for your response.
> > CDH is a Cloudera (third party) distribution. is there any to get the
> > notifications copy of it when cluster state changed ? in logs ?
> >
> > I can assume that the exception is result of no availability of replicas
> > only. Agree?
>
> Yes, I think that Solr believes there are no replicas for at least one
> shard.  As for why it believes that, I cannot say.
>
> If Solr logged every single thing that happened where zookeeper (or even
> just the clusterstate) is involved, you'd be drowning in logs.  Much
> more than already happens.  The logfile is already very verbose.
>
> Chances are that at least one of your Solr nodes *did* log something
> related to a problem with that collection before you got the error
> you're asking about.
>
> The "No live SolrServers" error is one that people are seeing quite
> frequently.  There may be some instances where Solr isn't behaving
> correctly, but I think when this happens, it usually indicates there's a
> real problem of some kind.
>
> To troubleshoot, we'll need to see any errors or warnings you find in
> your Solr logfiles from the time before you get an error on a request.
> You'll need to check the logfile on all Solr nodes.
>
> It might be a good idea to also involve Cloudera support, see what they
> think.
>
> Thanks,
> Shawn
>
>

Re: No live SolrServers available to handle this request

[Shawn Heisey]

On 3/17/2016 4:22 AM, Anil wrote:
> We are using solrcloud with zookeeper and each collection has 5 shareds and
> 2 replicas.
> we are seeing "org.apache.solr.client.solrj.SolrServerException: No live
> SolrServers available to handle this request". i dont see any issues with
> replicas.
>
> what would be root cause of the exception ? Thanks.

SolrCloud will determine what nodes/cores are available by looking at
the clusterstate.  The clusterstate might be indicating that all
replicas are down for one or more shards in your collection.

Exactly where the clusterstate lives in zookeeper will depend on what
version you are running and whether your collections on this cloud were
ever handled by 4.x versions of Solr.  Legacy collections will have
their clusterstate in /clusterstate.json, newer collections created by
5.x will have /collections//state.json, where  is the name of
the collection.  These are zookeeper paths, not filesystem paths.

Thanks,
Shawn

Re: No live SolrServers available to handle this request

[Shawn Heisey]

On 3/18/2016 9:55 PM, Anil wrote:
> Thanks for your response.
> CDH is a Cloudera (third party) distribution. is there any to get the
> notifications copy of it when cluster state changed ? in logs ?
>
> I can assume that the exception is result of no availability of replicas
> only. Agree?

Yes, I think that Solr believes there are no replicas for at least one
shard.  As for why it believes that, I cannot say.

If Solr logged every single thing that happened where zookeeper (or even
just the clusterstate) is involved, you'd be drowning in logs.  Much
more than already happens.  The logfile is already very verbose.

Chances are that at least one of your Solr nodes *did* log something
related to a problem with that collection before you got the error
you're asking about.

The "No live SolrServers" error is one that people are seeing quite
frequently.  There may be some instances where Solr isn't behaving
correctly, but I think when this happens, it usually indicates there's a
real problem of some kind.

To troubleshoot, we'll need to see any errors or warnings you find in
your Solr logfiles from the time before you get an error on a request. 
You'll need to check the logfile on all Solr nodes.

It might be a good idea to also involve Cloudera support, see what they
think.

Thanks,
Shawn

Re: No live SolrServers available to handle this request

[Anil]

HI Michael,

i could not post the query. i know its difficult to find out the root cause
without query. sorry about that.

query includes expand/collpase and query filter (fq) and 2 to 3 terms with
AND.

please share your thoughts. thanks.

Regards,
Anil

On 17 March 2016 at 19:46, michael solomon  wrote:

> What query do you try?
>
> On Thu, Mar 17, 2016 at 12:22 PM, Anil  wrote:
>
> > HI,
> >
> > We are using solrcloud with zookeeper and each collection has 5 shareds
> and
> > 2 replicas.
> > we are seeing "org.apache.solr.client.solrj.SolrServerException: No live
> > SolrServers available to handle this request". i dont see any issues with
> > replicas.
> >
> > what would be root cause of the exception ? Thanks.
> >
> > Regards,
> > Anil
> >
>

Re: No live SolrServers available to handle this request

[Anil]

Thanks Shawn. we are using 4.10.3.

I don't see any issues with replicas of all shards at the time of
exception. health of all shards is good in CDH.

Regards,
Anil



On 18 March 2016 at 10:52, Shawn Heisey  wrote:

> On 3/17/2016 4:22 AM, Anil wrote:
> > We are using solrcloud with zookeeper and each collection has 5 shareds
> and
> > 2 replicas.
> > we are seeing "org.apache.solr.client.solrj.SolrServerException: No live
> > SolrServers available to handle this request". i dont see any issues with
> > replicas.
> >
> > what would be root cause of the exception ? Thanks.
>
> SolrCloud will determine what nodes/cores are available by looking at
> the clusterstate.  The clusterstate might be indicating that all
> replicas are down for one or more shards in your collection.
>
> Exactly where the clusterstate lives in zookeeper will depend on what
> version you are running and whether your collections on this cloud were
> ever handled by 4.x versions of Solr.  Legacy collections will have
> their clusterstate in /clusterstate.json, newer collections created by
> 5.x will have /collections//state.json, where  is the name of
> the collection.  These are zookeeper paths, not filesystem paths.
>
> Thanks,
> Shawn
>
>

Re: No live SolrServers available to handle this request

[Shawn Heisey]

On 3/17/2016 11:29 PM, Anil wrote:
> Thanks Shawn. we are using 4.10.3.
>
> I don't see any issues with replicas of all shards at the time of
> exception. health of all shards is good in CDH.

I do not know what CDH is.  I'm guessing it's third-party software.  As
far as I'm aware, Solr doesn't have anything called CDH.

If you open Cloud->Tree in Solr's admin UI, you should be able to
examine the clusterstate.json file.  This will show you what Solr
believes about what parts of the cluster are up/down.

Thanks,
Shawn

Re: No live SolrServers available to handle this request

[michael solomon]

What query do you try?

On Thu, Mar 17, 2016 at 12:22 PM, Anil  wrote:

> HI,
>
> We are using solrcloud with zookeeper and each collection has 5 shareds and
> 2 replicas.
> we are seeing "org.apache.solr.client.solrj.SolrServerException: No live
> SolrServers available to handle this request". i dont see any issues with
> replicas.
>
> what would be root cause of the exception ? Thanks.
>
> Regards,
> Anil
>

Re: No live SolrServers available to handle this request

[Anil]

and defType is edismax

On 18 March 2016 at 10:40, Anil  wrote:

> HI Michael,
>
> i could not post the query. i know its difficult to find out the root
> cause without query. sorry about that.
>
> query includes expand/collpase and query filter (fq) and 2 to 3 terms with
> AND.
>
> please share your thoughts. thanks.
>
> Regards,
> Anil
>
> On 17 March 2016 at 19:46, michael solomon  wrote:
>
>> What query do you try?
>>
>> On Thu, Mar 17, 2016 at 12:22 PM, Anil  wrote:
>>
>> > HI,
>> >
>> > We are using solrcloud with zookeeper and each collection has 5 shareds
>> and
>> > 2 replicas.
>> > we are seeing "org.apache.solr.client.solrj.SolrServerException: No live
>> > SolrServers available to handle this request". i dont see any issues
>> with
>> > replicas.
>> >
>> > what would be root cause of the exception ? Thanks.
>> >
>> > Regards,
>> > Anil
>> >
>>
>
>

Re: No live SolrServers available to handle this request

[Anil]

HI Shawn,

Thanks for your response.
CDH is a Cloudera (third party) distribution. is there any to get the
notifications copy of it when cluster state changed ? in logs ?

I can assume that the exception is result of no availability of replicas
only. Agree?

Regards,
Anil

On 18 March 2016 at 18:20, Shawn Heisey  wrote:

> On 3/17/2016 11:29 PM, Anil wrote:
> > Thanks Shawn. we are using 4.10.3.
> >
> > I don't see any issues with replicas of all shards at the time of
> > exception. health of all shards is good in CDH.
>
> I do not know what CDH is.  I'm guessing it's third-party software.  As
> far as I'm aware, Solr doesn't have anything called CDH.
>
> If you open Cloud->Tree in Solr's admin UI, you should be able to
> examine the clusterstate.json file.  This will show you what Solr
> believes about what parts of the cluster are up/down.
>
> Thanks,
> Shawn
>
>

Re: No live SolrServers available to handle this request

[wilanjar .]

Hi Erick,

Thanks for your response.
You right my node running properly and the graph is green.
we solve with remove the data index in collection and reindex again.

Thanks

On Fri, Nov 6, 2015 at 11:02 PM, Erick Erickson 
wrote:

> The host may be running well, but my bet is that
> you have an error in the schema.xml file so it's
> no longer valid XML and the core did not load.
>
> So while the solr instance is up and running, no
> core using that schema is running, thus no
> live servers.
>
> Look at the admin UI, cloud>>graph view and
> if the collection you're trying to operate on is
> not green, then that's probably the issue.
>
> Otherwise look through the Solr log file and
> you should see some exceptions that may
> point the way.
>
> Best,
> Erick
>
> On Thu, Nov 5, 2015 at 11:58 PM, wilanjar .  wrote:
> > Hi All,
> >
> > I'm very new handle the solrcloud.
> > I've changed the scema.xml with adding field to index but after reload
> the
> > collection we got error from logging " No live SolrServers available to
> > handle this request".
> >
> > i have check solrcloud from localhost each node and running  well.
> > i'm using solr version 4.10.4 lucene version 4.10.4
> > tomcat 8.0.27
> > zookeeper 3.4.6.
> >
> > I already googling but not get solution yet.
> >
> > Thank you.
>

Re: No live SolrServers available to handle this request

[Erick Erickson]

The host may be running well, but my bet is that
you have an error in the schema.xml file so it's
no longer valid XML and the core did not load.

So while the solr instance is up and running, no
core using that schema is running, thus no
live servers.

Look at the admin UI, cloud>>graph view and
if the collection you're trying to operate on is
not green, then that's probably the issue.

Otherwise look through the Solr log file and
you should see some exceptions that may
point the way.

Best,
Erick

On Thu, Nov 5, 2015 at 11:58 PM, wilanjar .  wrote:
> Hi All,
>
> I'm very new handle the solrcloud.
> I've changed the scema.xml with adding field to index but after reload the
> collection we got error from logging " No live SolrServers available to
> handle this request".
>
> i have check solrcloud from localhost each node and running  well.
> i'm using solr version 4.10.4 lucene version 4.10.4
> tomcat 8.0.27
> zookeeper 3.4.6.
>
> I already googling but not get solution yet.
>
> Thank you.

Re: No live SolrServers available to handle this request

[Steve]

Thanks Mark,

I rebuilt and made sure the versions matched.  It works.
Not sure how that happened tho..

thx.
.strick

On Thu, Oct 8, 2015 at 4:31 PM, Mark Miller  wrote:

> Your Lucene and Solr versions must match.
>
> On Thu, Oct 8, 2015 at 4:02 PM Steve  wrote:
>
> > I've loaded the Films data into a 4 node cluster.  Indexing went well,
> but
> > when I issue a query, I get this:
> >
> > "error": {
> > "msg": "org.apache.solr.client.solrj.SolrServerException: No live
> > SolrServers available to handle this request:
> > [
> >
> >
> http://host-192-168-0-63.openstacklocal:8081/solr/CollectionFilms_shard1_replica2
> > ,
> >
> >
> >
> http://host-192-168-0-62.openstacklocal:8081/solr/CollectionFilms_shard2_replica2
> > ,
> >
> >
> >
> http://host-192-168-0-60.openstacklocal:8081/solr/CollectionFilms_shard2_replica1
> > ]",
> > ...
> >
> > and further down in the stacktrace:
> >
> > Server Error
> > Caused by:
> > java.lang.NoSuchMethodError:
> >
> >
> org.apache.lucene.index.TermsEnum.postings(Lorg/apache/lucene/index/PostingsEnum;I)Lorg/apache/lucene/index/PostingsEnum;\n\tat
> >
> >
> org.apache.solr.search.SolrIndexSearcher.getFirstMatch(SolrIndexSearcher.java:802)\n\tat
> >
> >
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:333)\n\tat
> > ...
> >
> >
> > I'm using:
> >
> > solr version 5.3.1
> >
> > lucene 5.2.1
> >
> > zookeeper version 3.4.6
> >
> > indexing with:
> >
> >cd /opt/solr/example/films;
> >
> > /opt/solr/bin/post -c CollectionFilms -port 8081  films.json
> >
> >
> >
> > thx,
> > .strick
> >
> --
> - Mark
> about.me/markrmiller
>

Re: No live SolrServers available to handle this request

[Mark Miller]

Your Lucene and Solr versions must match.

On Thu, Oct 8, 2015 at 4:02 PM Steve  wrote:

> I've loaded the Films data into a 4 node cluster.  Indexing went well, but
> when I issue a query, I get this:
>
> "error": {
> "msg": "org.apache.solr.client.solrj.SolrServerException: No live
> SolrServers available to handle this request:
> [
>
> http://host-192-168-0-63.openstacklocal:8081/solr/CollectionFilms_shard1_replica2
> ,
>
>
> http://host-192-168-0-62.openstacklocal:8081/solr/CollectionFilms_shard2_replica2
> ,
>
>
> http://host-192-168-0-60.openstacklocal:8081/solr/CollectionFilms_shard2_replica1
> ]",
> ...
>
> and further down in the stacktrace:
>
> Server Error
> Caused by:
> java.lang.NoSuchMethodError:
>
> org.apache.lucene.index.TermsEnum.postings(Lorg/apache/lucene/index/PostingsEnum;I)Lorg/apache/lucene/index/PostingsEnum;\n\tat
>
> org.apache.solr.search.SolrIndexSearcher.getFirstMatch(SolrIndexSearcher.java:802)\n\tat
>
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:333)\n\tat
> ...
>
>
> I'm using:
>
> solr version 5.3.1
>
> lucene 5.2.1
>
> zookeeper version 3.4.6
>
> indexing with:
>
>cd /opt/solr/example/films;
>
> /opt/solr/bin/post -c CollectionFilms -port 8081  films.json
>
>
>
> thx,
> .strick
>
-- 
- Mark
about.me/markrmiller