[
https://issues.apache.org/jira/browse/SOLR-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vadym Lotar updated SOLR-5364:
------------------------------
Attachment: threaddump.201311180812.zip
SOLR got stuck again both on Staging and Production environment. I attached
thread dump taken from the production env.
Here is the main interesting part:
Thread 14813: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may
be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=186 (Compiled frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=834 (Compiled frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int)
@bci=72, line=994 (Compiled frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int)
@bci=24, line=1303 (Compiled frame)
- java.util.concurrent.Semaphore.acquire() @bci=5, line=317 (Compiled frame)
- org.apache.solr.util.AdjustableSemaphore.acquire() @bci=4, line=61 (Compiled
frame)
-
org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.update.SolrCmdDistributor$Request)
@bci=22, line=418 (Compiled frame)
-
org.apache.solr.update.SolrCmdDistributor.submit(org.apache.solr.client.solrj.request.UpdateRequest,
org.apache.solr.update.SolrCmdDistributor$Node) @bci=20, line=368 (Compiled
frame)
- org.apache.solr.update.SolrCmdDistributor.flushAdds(int) @bci=223, line=300
(Compiled frame)
- org.apache.solr.update.SolrCmdDistributor.finish() @bci=2, line=96 (Compiled
frame)
- org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish()
@bci=4, line=499 (Compiled frame)
- org.apache.solr.update.processor.DistributedUpdateProcessor.finish() @bci=8,
line=1256 (Compiled frame)
- org.apache.solr.update.processor.LogUpdateProcessor.finish() @bci=48,
line=179 (Compiled frame)
-
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=155, line=83 (Compiled frame)
-
org.apache.solr.handler.RequestHandlerBase.handleRequest(org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=43, line=135 (Compiled frame)
-
org.apache.solr.core.SolrCore.execute(org.apache.solr.request.SolrRequestHandler,
org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=115, line=1859 (Compiled frame)
-
org.apache.solr.servlet.SolrDispatchFilter.execute(javax.servlet.http.HttpServletRequest,
org.apache.solr.request.SolrRequestHandler,
org.apache.solr.request.SolrQueryRequest,
org.apache.solr.response.SolrQueryResponse) @bci=31, line=703 (Compiled frame)
-
org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse, javax.servlet.FilterChain, boolean) @bci=1348,
line=406 (Interpreted frame)
-
org.apache.solr.servlet.SolrDispatchFilter.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=5, line=195
(Compiled frame)
-
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(javax.servlet.ServletRequest,
javax.servlet.ServletResponse) @bci=100, line=1419 (Interpreted frame)
- org.eclipse.jetty.servlet.ServletHandler.doHandle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=197, line=455 (Interpreted frame)
- org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=25, line=137 (Compiled frame)
- org.eclipse.jetty.security.SecurityHandler.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=540, line=557 (Compiled frame)
- org.eclipse.jetty.server.session.SessionHandler.doHandle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=68, line=231 (Interpreted frame)
- org.eclipse.jetty.server.handler.ContextHandler.doHandle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=206, line=1075 (Interpreted frame)
- org.eclipse.jetty.servlet.ServletHandler.doScope(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=277, line=384 (Interpreted frame)
- org.eclipse.jetty.server.session.SessionHandler.doScope(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=233, line=193 (Interpreted frame)
- org.eclipse.jetty.server.handler.ContextHandler.doScope(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=437, line=1009 (Interpreted frame)
- org.eclipse.jetty.server.handler.ScopedHandler.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=13, line=135 (Compiled frame)
-
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=399, line=255 (Compiled frame)
- org.eclipse.jetty.server.handler.HandlerCollection.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=42, line=154 (Compiled frame)
- org.eclipse.jetty.server.handler.HandlerWrapper.handle(java.lang.String,
org.eclipse.jetty.server.Request, javax.servlet.http.HttpServletRequest,
javax.servlet.http.HttpServletResponse) @bci=23, line=116 (Compiled frame)
-
org.eclipse.jetty.server.Server.handle(org.eclipse.jetty.server.AbstractHttpConnection)
@bci=146, line=368 (Compiled frame)
- org.eclipse.jetty.server.AbstractHttpConnection.handleRequest() @bci=323,
line=489 (Compiled frame)
- org.eclipse.jetty.server.BlockingHttpConnection.handleRequest() @bci=1,
line=53 (Compiled frame)
-
org.eclipse.jetty.server.AbstractHttpConnection.content(org.eclipse.jetty.io.Buffer)
@bci=13, line=953 (Compiled frame)
-
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(org.eclipse.jetty.io.Buffer)
@bci=5, line=1014 (Compiled frame)
- org.eclipse.jetty.http.HttpParser.parseNext() @bci=2044, line=622 (Compiled
frame)
- org.eclipse.jetty.http.HttpParser.parseAvailable() @bci=51, line=240
(Compiled frame)
- org.eclipse.jetty.server.BlockingHttpConnection.handle() @bci=51, line=72
(Interpreted frame)
- org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run()
@bci=100, line=264 (Interpreted frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(java.lang.Runnable)
@bci=1, line=608 (Interpreted frame)
- org.eclipse.jetty.util.thread.QueuedThreadPool$3.run() @bci=47, line=543
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=724 (Interpreted frame)
So, it seems like synchronization issue inside of SOLR. Today I'm gonna go
deeply inside the SOLR code and most likely try to check Chris's suggestion
regarding WhitespaceTokenizer.
> SolrCloud stops accepting updates
> ---------------------------------
>
> Key: SOLR-5364
> URL: https://issues.apache.org/jira/browse/SOLR-5364
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.4, 4.5, 4.6
> Reporter: Chris
> Priority: Blocker
> Attachments: threaddump.201311180812.zip
>
>
> I'm attempting to import data into a SolrCloud cluster. After a certain
> amount of time, the cluster stops accepting updates.
> I have tried numerous suggestions in IRC from Elyorag and others without
> resolve.
> I have had this issue with 4.4, and understood there was a deadlock issue
> fixed in 4.5, which hasn't resolved the issue, neither have the 4.6 snapshots.
> I've tried with Tomcat, various tomcat configuration changes to threading,
> and with Jetty. Tried with various index merging configurations as I
> initially thought there was a deadlock with concurrent merg scheduler,
> however same issue with SerialMergeScheduler.
> The cluster stops accepting updates after some amount of time, this seems to
> vary and is inconsistent. Sometimes I manage to index 400k docs, other times
> ~1million . Querying the cluster continues to work. I can reproduce the
> issue consistently, and is currently blocking our transition to Solr.
> I can provide stack traces, thread dumps, jstack dumps as required.
> Here are two jstacks thus far:
> http://pastebin.com/1ktjBYbf
> http://pastebin.com/8JiQc3rb
> I have got these jstacks from the latest 4.6 snapshot, also running solrj
> snapshot. The issue is also consistently reproducable with BinaryRequest
> writer.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
