David Johnson created SOLR-10917:
------------------------------------
Summary: Port starvation issue in Luke
Key: SOLR-10917
URL: https://issues.apache.org/jira/browse/SOLR-10917
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: Admin UI, SolrCloud
Affects Versions: 6.3
Reporter: David Johnson
If a collection replicated in multiple nodes in a cluster is completely down a
request to Luke for that collection can sometimes result in a port starvation
issue as the request is handed around the cluster and rejected by each node.
We experienced an issue in which 10000 requests were made (3300 per node) in
under a second, rendering the entire cluster unresponsive - this required a
full restart of all nodes in the cluster to restore service.
Logged connections were all similar to:
2017-06-19 17:54:51.461 ERROR (qtp606548741-15083) [ ] o.a.s.s.HttpSolrCall
null:org.apache.solr.common.SolrException: Error trying to proxy request for
url: http://redacted/solr/collection/config
at
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:590)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:303)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.eclipse.jetty.io.EofException
at
org.eclipse.jetty.server.HttpConnection$SendCallback.reset(HttpConnection.java:666)
at
org.eclipse.jetty.server.HttpConnection$SendCallback.access$300(HttpConnection.java:630)
at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:511)
at
org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:668)
at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:722)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:179)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:163)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:415)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2147)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:2102)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:2123)
at
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:586)
... 27 more
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
