Re: Proxy Error when cluster went down

[Vishal Vaibhav]

So entire cluster was down. I m trying to bring node by node . I restarted
the first node . The solr comes up but add replicas command fails. And then
I tried to check clusterstatus api, it showed shard in active state, no
core as active i.e. all down and one live node which was the one that I
restarted. Also this all connects to one zookeeper ensemble of 3 nodes

On Tue, 16 Jun 2020 at 11:20 PM, Jörn Franke  wrote:

> Do you have another host with replica alive or are all replicas on the
> host that is down?
>
> Are all SolrCloud hosts in the same ZooKeeper?
>
> > Am 16.06.2020 um 19:29 schrieb Vishal Vaibhav :
> >
> > Hi thanks . My solr is running in kubernetes. So host name goes away
> with
> > the pod going
> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.s
> > vc.cluster.local
> >  So in my case the pod with this host has gone and also the hostname
> >
> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local
> > is no more there.. should not the solr cloud be aware of the fact that
> all
> > the replicas in that solr host is down and should not proxy the request
> to
> > that node..
> >
> >> On Tue, 16 Jun 2020 at 5:06 PM, Shawn Heisey 
> wrote:
> >>
> >>> On 6/15/2020 9:04 PM, Vishal Vaibhav wrote:
> >>> I am running on solr 8.5. For some reason entire cluster went down.
> When
> >> i
> >>> am trying to bring up the nodes,its not coming up. My health check is
> >>> on "/solr/rules/admin/system". I tried forcing a leader election but it
> >>> dint help.
> >>> so when i run the following commands. Why is it trying to proxy when
> >> those
> >>> nodes are down. Am i missing something?
> >>
> >> 
> >>
> >>> java.net.UnknownHostException:
> >>>
> >>
> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local:
> >>
> >> It is trying to proxy because it's SolrCloud.  SolrCloud has an internal
> >> load balancer that spreads queries across multiple replicas when
> >> possible.  Your cluster must be aware of multiple servers where the
> >> "rules" collection can be queried.
> >>
> >> The underlying problem behind this error message is that the following
> >> hostname is being looked up, and it doesn't exist:
> >>
> >>
> >>
> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local
> >>
> >> This hostname is most likely coming from /etc/hosts on one of your
> >> systems when that system starts Solr and it registers with the cluster,
> >> and that /etc/hosts file is the ONLY place that the hostname exists, so
> >> when SolrCloud tries to forward the request to that server, it is
> failing.
> >>
> >> Thanks,
> >> Shawn
> >>
>

Re: Proxy Error when cluster went down

[Jörn Franke]

Do you have another host with replica alive or are all replicas on the host 
that is down?

Are all SolrCloud hosts in the same ZooKeeper?

> Am 16.06.2020 um 19:29 schrieb Vishal Vaibhav :
> 
> Hi thanks . My solr is running in kubernetes. So host name goes away with
> the pod going search-rules-solr-v1-2.search-rules-solr-v1.search-digital.s
> vc.cluster.local
>  So in my case the pod with this host has gone and also the hostname
> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local
> is no more there.. should not the solr cloud be aware of the fact that all
> the replicas in that solr host is down and should not proxy the request to
> that node..
> 
>> On Tue, 16 Jun 2020 at 5:06 PM, Shawn Heisey  wrote:
>> 
>>> On 6/15/2020 9:04 PM, Vishal Vaibhav wrote:
>>> I am running on solr 8.5. For some reason entire cluster went down. When
>> i
>>> am trying to bring up the nodes,its not coming up. My health check is
>>> on "/solr/rules/admin/system". I tried forcing a leader election but it
>>> dint help.
>>> so when i run the following commands. Why is it trying to proxy when
>> those
>>> nodes are down. Am i missing something?
>> 
>> 
>> 
>>> java.net.UnknownHostException:
>>> 
>> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local:
>> 
>> It is trying to proxy because it's SolrCloud.  SolrCloud has an internal
>> load balancer that spreads queries across multiple replicas when
>> possible.  Your cluster must be aware of multiple servers where the
>> "rules" collection can be queried.
>> 
>> The underlying problem behind this error message is that the following
>> hostname is being looked up, and it doesn't exist:
>> 
>> 
>> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local
>> 
>> This hostname is most likely coming from /etc/hosts on one of your
>> systems when that system starts Solr and it registers with the cluster,
>> and that /etc/hosts file is the ONLY place that the hostname exists, so
>> when SolrCloud tries to forward the request to that server, it is failing.
>> 
>> Thanks,
>> Shawn
>> 

Re: Proxy Error when cluster went down

[Vishal Vaibhav]

Hi thanks . My solr is running in kubernetes. So host name goes away with
the pod going search-rules-solr-v1-2.search-rules-solr-v1.search-digital.s
vc.cluster.local
  So in my case the pod with this host has gone and also the hostname
search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local
is no more there.. should not the solr cloud be aware of the fact that all
the replicas in that solr host is down and should not proxy the request to
that node..

On Tue, 16 Jun 2020 at 5:06 PM, Shawn Heisey  wrote:

> On 6/15/2020 9:04 PM, Vishal Vaibhav wrote:
> > I am running on solr 8.5. For some reason entire cluster went down. When
> i
> > am trying to bring up the nodes,its not coming up. My health check is
> > on "/solr/rules/admin/system". I tried forcing a leader election but it
> > dint help.
> > so when i run the following commands. Why is it trying to proxy when
> those
> > nodes are down. Am i missing something?
>
> 
>
> > java.net.UnknownHostException:
> >
> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local:
>
> It is trying to proxy because it's SolrCloud.  SolrCloud has an internal
> load balancer that spreads queries across multiple replicas when
> possible.  Your cluster must be aware of multiple servers where the
> "rules" collection can be queried.
>
> The underlying problem behind this error message is that the following
> hostname is being looked up, and it doesn't exist:
>
>
> search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local
>
> This hostname is most likely coming from /etc/hosts on one of your
> systems when that system starts Solr and it registers with the cluster,
> and that /etc/hosts file is the ONLY place that the hostname exists, so
> when SolrCloud tries to forward the request to that server, it is failing.
>
> Thanks,
> Shawn
>

Re: Proxy Error when cluster went down

[Shawn Heisey]

On 6/15/2020 9:04 PM, Vishal Vaibhav wrote:

I am running on solr 8.5. For some reason entire cluster went down. When i
am trying to bring up the nodes,its not coming up. My health check is
on "/solr/rules/admin/system". I tried forcing a leader election but it
dint help.
so when i run the following commands. Why is it trying to proxy when those
nodes are down. Am i missing something?





java.net.UnknownHostException:
search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local:


It is trying to proxy because it's SolrCloud.  SolrCloud has an internal 
load balancer that spreads queries across multiple replicas when 
possible.  Your cluster must be aware of multiple servers where the 
"rules" collection can be queried.


The underlying problem behind this error message is that the following 
hostname is being looked up, and it doesn't exist:


search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local

This hostname is most likely coming from /etc/hosts on one of your 
systems when that system starts Solr and it registers with the cluster, 
and that /etc/hosts file is the ONLY place that the hostname exists, so 
when SolrCloud tries to forward the request to that server, it is failing.


Thanks,
Shawn

Proxy Error when cluster went down

[Vishal Vaibhav]

Hello all,

I am running on solr 8.5. For some reason entire cluster went down. When i
am trying to bring up the nodes,its not coming up. My health check is
on "/solr/rules/admin/system". I tried forcing a leader election but it
dint help.
so when i run the following commands. Why is it trying to proxy when those
nodes are down. Am i missing something?


*curl "http://localhost:8983/solr/rules/admin/system
" * % Total% Received %
Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft
 Speed
100  5582  100  55820 0   454k  0 --:--:-- --:--:-- --:--:--
 495k
{
  "error":{
"metadata":[
  "error-class","org.apache.solr.common.SolrException",
  "root-error-class","java.net.UnknownHostException"],
"msg":"Error trying to proxy request for url:
http://search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local:8983/solr/rules/admin/system
",
"trace":"org.apache.solr.common.SolrException: Error trying to proxy
request for url:
http://search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local:8983/solr/rules/admin/system\n\tat
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:735)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:562)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:505)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)\n\tat
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917)\n\tat
java.base/java.lang.Thread.run(Unknown Source)\nCaused by:
java.net.UnknownHostException:
search-rules-solr-v1-2.search-rules-solr-v1.search-digital.svc.cluster.local:
Name or service not known\n\tat
java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)\n\tat
java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(Unknown
Source)\n\tat
java.base/java.net.InetAddress.getAddressesFromNameService(Unknown
Source)\n\tat
java.base/java.net.InetAddress$NameServiceAddresses.get(Unknown
Source)\n\tat java.base/java.net.InetAddress.getAllByName0(Unknown