Alapan Sur wrote:
Hi Team,
We have .Net application consuming tomcat solr3.5 services and deployed on
Windows Server 2012. Its is 64 bit machine with 32 GB RAM and Tomcat veriosn
is 6 and is installed at :"C:\Program Files (x86)\Apache Software
Foundation\Tomcat 6.0". During the peak load, the solr sevice
stops/unresponsive and it is happening very frequently. We didnt fount any
erro logs in windows event viewer but in tomcat logs we got the following
exception :
Aug 06, 2014 10:51:18 AM org.apache.solr.common.SolrException log SEVERE:
ClientAbortException: java.net.SocketException: Software caused connection
abort: socket write error at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:
369) at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:339) at
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:392)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:381)
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.ja
va:89) at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) at
sun.nio.cs.StreamEncoder.implWrite(Unknown Source) at
sun.nio.cs.StreamEncoder.write(Unknown Source) at
java.io.OutputStreamWriter.write(Unknown Source) at
org.apache.solr.common.util.FastWriter.write(FastWriter.java:55) at
org.apache.solr.common.util.FastWriter.write(FastWriter.java:50) at
org.apache.solr.response.XMLWriter.startTag(XMLWriter.java:269) at
org.apache.solr.response.XMLWriter.writePrim(XMLWriter.java:780) at
org.apache.solr.response.XMLWriter.writeInt(XMLWriter.java:691) at
org.apache.solr.response.XMLWriter.writeVal(XMLWriter.java:565) at
org.apache.solr.response.XMLWriter.writeNamedList(XMLWriter.java:621) at
org.apache.solr.response.XMLWriter.writeVal(XMLWriter.java:594) at
org.apache.solr.response.XMLWriter.writeNamedList(XMLWriter.java:621) at
org.apache.solr.response.XMLWriter.writeVal(XMLWriter.java:594) at
org.apache.solr.response.XMLWriter.writeNamedList(XMLWriter.java:621) at
org.apache.solr.response.XMLWriter.writeVal(XMLWriter.java:594) at
org.apache.solr.response.XMLWriter.writeResponse(XMLWriter.java:132) at
org.apache.solr.response.XMLResponseWriter.write(XMLResponseWriter.java:35)
at
org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.
java:343) at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
265) at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:235) at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206) at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:233) at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:191) at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127
) at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103
) at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:109) at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http
11Protocol.java:606) at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at
java.lang.Thread.run(Unknown Source) Caused by: java.net.SocketException:
Software caused connection abort: socket write error at
java.net.SocketOutputStream.socketWrite0(Native Method) at
java.net.SocketOutputStream.socketWrite(Unknown Source) at
java.net.SocketOutputStream.write(Unknown Source) at
org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputB
uffer.java:761) at
org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:448) at
org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:363) at
org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWri
te(InternalOutputBuffer.java:785) at
org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFi
lter.java:126) at
org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.j
ava:598) at org.apache.coyote.Response.doWrite(Response.java:560) at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:
364) ... 36 more
Aug 06, 2014 10:51:29 AM org.apache.solr.common.SolrException log SEVERE:
java.lang.OutOfMemoryError: Java heap space at
org.apache.lucene.search.ExactPhraseScorer.(ExactPhraseScorer.java:38) at
org.apache.lucene.search.PhraseQuery$PhraseWeight.scorer(PhraseQuery.java:22
7) at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java
:298) at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:577) at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:364) at
org.apache.solr.search.SolrIndexSearcher.getDocListAndSetNC(SolrIndexSearche
r.java:1408) at
org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:
1158) at
org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:362)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java
:378) at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHand
ler.java:194) at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.
java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1372) at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:3
56) at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:
252) at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:235) at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206) at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:233) at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:191) at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127
) at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103
) at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:109) at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http
11Protocol.java:606) at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at
java.lang.Thread.run(Unknown Source)
Aug 07, 2014 11:53:10 AM org.apache.tomcat.util.net.JIoEndpoint
createWorkerThread INFO: Maximum number of threads (200) created for
connector with address null and port 8080
Can anyone help on this error ?
Maybe.
Let's start with the last one : that is an "INFO" message, not an error. It tells you
that this <Connector port="8080" ...>, which has a (probably default) configuration with
the maxThreads attribute set to 200, has reached this value, and will not start any more
additional threads to process new requests. Which means that apparently, there are
already 200 threads currently running and busy processing client requests at this time.
This could explain why the server is now "unresponsive" : after a while (as its "accept
queue" fills up), it will stop accepting new client connections, as long as not at least
one of these busy threads finishes what it is doing, and becomes available to process
another client request.
Unfortunately the previous exeception (> Aug 06, 2014 10:51:29 AM
org.apache.solr.common.SolrException log SEVERE:
> java.lang.OutOfMemoryError: Java heap space at..)
kind of muddies the waters, because after an out-of-memory error, many strange and
wonderful things can happen.
Including maybe 200 threads which would appear to be running and processing client
requests, but in reality are not doing anything useful anymore.
In your description, you mention that the machine is 64-bit and has 32 GB RAM.
How much of that RAM is really available to the Java JVM which runs Tomcat ? And what are
the settings for the Heap size of Java/Tomcat ?
To check this, run the "tomcat6w.exe" program which comes with Tomcat (in
...(tomcat_dir)/bin), and have a look at the java tab, which tells you what the
"Initial/maximum memory pool" settings say.
You are running Solr/Lucene, which is a full-text search engine. I am not a specialist of
Solr/Lucene per se, but search engines in general tend to use a *lot* of memory to store
intermediate search results, if the indexes being managed are large (aka the searched
collections are large).
And, also in general, they tend to have a poor failover behaviour when they do not have
enough memory to handle some search requests.
In the absence of any more information, and just making a somewhat uneducated guess, I
would tend to say that anything less than 1 GB for the java "memory pool" (the Heap), is
probably inadequate, unless this is just a small test system with a small collection of
indexed items.
Note : if my memory serves right, Solr is the part which runs under Tomcat, and it is a
front-end to Lucene, which is probably running in a separate Java process.
And I don't know who here is really using up the memory. But your "out of Heap space"
above is happening in Tomcat, so I'd concentrate on that first, and see whether getting
rid of that problem improves the situation.
Ah, and by the way, tomcat 6 is rather old and outdated by now. You may want to upgrade
it to a more recent version (8, or at least 7). That would get you better support in the
future.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
