Re: Solr4.7 No live SolrServers available to handle this request

Fri, 21 Mar 2014 08:01:45 -0700

I just managed to track this down -- as you said the disconnect was a red herring.
Ultimately the problem was caused by a custom analysis component we wrote that was raising an IOException -- it was missing some configuration files it relies on. 


What might be interesting for solr devs to have a look at is that 
exception was completely swallowed by JavabinCodec, making it very 
difficult to track down the problem.  Furthermore -- if the /add request 
was routed directly to the shard where the document was destined to end 
up, then the IOException raised by the analysis component (a char 
filter) showed up in the Solr HTTP response (probably because my client 
used XML format in one test -- javabin is used internally in 
SolrCloud).  But if the request was routed to a different shard, then 
the only exception that showed up anywhere (in the logs, in the HTTP 
response) was kind of irrelevant.


I think this could be fixed pretty easily; see SOLR-5985 for my suggestion.

-Mike


On 03/21/2014 10:20 AM, Greg Walters wrote:

Broken pipe errors are generally caused by unexpected disconnections and are some times 
hard to track down. Given the stack traces you've provided it's hard to point to any one 
thing and I suspect the relevant information was snipped out in the "long dump of 
document fields". You might grab the entire error from the client you're uploading 
documents with, the server you're connected to and any other nodes that have an error at 
the same time and put it on pastebin or the like.

Thanks,
Greg

On Mar 20, 2014, at 3:36 PM, Michael Sokolov <msoko...@safaribooksonline.com> 
wrote:


I'm getting a similar exception when writing documents (on the client side).  I 
can write one document fine, but the second (which is being routed to a 
different shard) generates the error.  It happens every time - definitely not a 
resource issue or timing problem since this database is completely empty -- I'm 
just getting started and running some tests, so there must be some kind of 
setup problem.  But it's difficult to diagnose (for me, anyway)!  I'd 
appreciate any insight, hints, guesses, etc. since I'm stuck. Thanks!

One node (the leader?) is reporting "Internal Server Error" in its log, and 
another node (presumably the shard where the document is being directed) bombs out like 
this:

ERROR - 2014-03-20 15:56:53.022; org.apache.solr.common.SolrException; 
null:org.apache.solr.common.SolrException: ERROR adding document 
SolrInputDocument(

... long dump of document fields

)
    at 
org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:99)
    at 
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:166)
    at 
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
    at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:225)
    at 
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:121)
    at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:190)
    at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:116)
    at 
org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:173)
    at 
org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:106)
    at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:58)
    at 
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
    at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
    at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859)
    at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721)
...
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
        at 
org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366)
        at 
org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:240)
        at 
org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:119)
        at 
org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192)
        at org.apache.coyote.Response.doWrite(Response.java:520)
        at 
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:408)
        ... 37 more

This is with Solr 4.6.1, Tomcat 7.  Here's my clusterstate.json. Updates are 
being sent to the test1x3 collection


{
  "test3x1":{
    "shards":{
      "shard1":{
        "range":"80000000-d554ffff",
        "state":"active",
        "replicas":{"core_node1":{
            "state":"active",
            "base_url":"http://10.4.24.37:8080/solr";,
            "core":"test3x1_shard1_replica1",
            "node_name":"10.4.24.37:8080_solr",
            "leader":"true"}}},
      "shard2":{
        "range":"d5550000-2aa9ffff",
        "state":"active",
        "replicas":{"core_node3":{
            "state":"active",
            "base_url":"http://10.4.24.39:8080/solr";,
            "core":"test3x1_shard2_replica1",
            "node_name":"10.4.24.39:8080_solr",
            "leader":"true"}}},
      "shard3":{
        "range":"2aaa0000-7fffffff",
        "state":"active",
        "replicas":{"core_node2":{
            "state":"active",
            "base_url":"http://10.4.24.38:8080/solr";,
            "core":"test3x1_shard3_replica1",
            "node_name":"10.4.24.38:8080_solr",
            "leader":"true"}}}},
    "maxShardsPerNode":"1",
    "router":{"name":"compositeId"},
    "replicationFactor":"1"},
  "test1x3":{
    "shards":{"shard1":{
        "range":"80000000-7fffffff",
        "state":"active",
        "replicas":{
          "core_node1":{
            "state":"active",
            "base_url":"http://10.4.24.39:8080/solr";,
            "core":"test1x3_shard1_replica2",
            "node_name":"10.4.24.39:8080_solr",
            "leader":"true"},
          "core_node2":{
            "state":"active",
            "base_url":"http://10.4.24.38:8080/solr";,
            "core":"test1x3_shard1_replica1",
            "node_name":"10.4.24.38:8080_solr"},
          "core_node3":{
            "state":"active",
            "base_url":"http://10.4.24.37:8080/solr";,
            "core":"test1x3_shard1_replica3",
            "node_name":"10.4.24.37:8080_solr"}}}},
    "maxShardsPerNode":"1",
    "router":{"name":"compositeId"},
    "replicationFactor":"3"},
  "test2x2":{
    "shards":{
      "shard1":{
        "range":"80000000-ffffffff",
        "state":"active",
        "replicas":{
          "core_node1":{
            "state":"active",
            "base_url":"http://10.4.24.39:8080/solr";,
            "core":"test2x2_shard1_replica1",
            "node_name":"10.4.24.39:8080_solr"},
          "core_node4":{
            "state":"active",
            "base_url":"http://10.4.24.38:8080/solr";,
            "core":"test2x2_shard1_replica2",
            "node_name":"10.4.24.38:8080_solr",
            "leader":"true"}}},
      "shard2":{
        "range":"0-7fffffff",
        "state":"active",
        "replicas":{
          "core_node2":{
            "state":"active",
            "base_url":"http://10.4.24.37:8080/solr";,
            "core":"test2x2_shard2_replica1",
            "node_name":"10.4.24.37:8080_solr",
            "leader":"true"},
          "core_node3":{
            "state":"active",
            "base_url":"http://10.4.24.39:8080/solr";,
            "core":"test2x2_shard2_replica2",
            "node_name":"10.4.24.39:8080_solr"}}}},
    "maxShardsPerNode":"2",
    "router":{"name":"compositeId"},
    "replicationFactor":"2"}}



On 03/20/2014 09:44 AM, Greg Walters wrote:

Sathya,

I assume you're using Solr Cloud. Please provide your clusterstate.json while 
you're seeing this issue and check your logs for any exceptions. With no 
information from you it's hard to troubleshoot any issues!

Thanks,
Greg

On Mar 20, 2014, at 12:44 AM, Sathya <sathia.blacks...@gmail.com> wrote:


Hi Friends,

I am new to Solr. I have 5 solr node in 5 different machine. When i index
the data, sometimes "*No live SolrServers available to handle this request*"
exception occur in 1 or 2 machines.

I dont know why its happen and how to solve this. Kindly help me to solve
this issue.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr4-7-No-live-SolrServers-available-to-handle-this-request-tp4125679.html
Sent from the Solr - User mailing list archive at Nabble.com.
























					Reply via email to