[
https://issues.apache.org/jira/browse/SOLR-9741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Forest Soup updated SOLR-9741:
------------------------------
Description:
When we doing a batch of index and search operations to SolrCloud v5.3.2, we
usually met a CPU% spike lasting about 10 min.
We have 5 physical servers, 2 solr instances running on each server with
different port(8983 and 8984), all 8983 are in a same solrcloud, all 8984 are
in another solrcloud.
You can see the chart in the attach file screenshot-1.png.
The thread dump are in the attach file threads.zip.
During the spike, the thread dump shows most of the threads are with the call
stacks below:
"qtp634210724-4759" #4759 prio=5 os_prio=0 tid=0x00007fb32803e000 nid=0x64e7
runnable [0x00007fb3ef1ef000]
java.lang.Thread.State: RUNNABLE
at
java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444)
at java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419)
at java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298)
at java.lang.ThreadLocal.get(ThreadLocal.java:163)
at
org.apache.solr.search.SolrQueryTimeoutImpl.get(SolrQueryTimeoutImpl.java:49)
at
org.apache.solr.search.SolrQueryTimeoutImpl.shouldExit(SolrQueryTimeoutImpl.java:57)
at
org.apache.lucene.index.ExitableDirectoryReader$ExitableTermsEnum.checkAndThrow(ExitableDirectoryReader.java:165)
at
org.apache.lucene.index.ExitableDirectoryReader$ExitableTermsEnum.<init>(ExitableDirectoryReader.java:157)
at
org.apache.lucene.index.ExitableDirectoryReader$ExitableTerms.iterator(ExitableDirectoryReader.java:141)
at org.apache.lucene.index.TermContext.build(TermContext.java:93)
at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:192)
at
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:855)
at org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:56)
at
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:203)
at
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:855)
at org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:56)
at
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:203)
at
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:855)
at org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:56)
at
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:203)
at
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:855)
at
org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:838)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:486)
at
org.apache.solr.search.Grouping.searchWithTimeLimiter(Grouping.java:456)
at org.apache.solr.search.Grouping.execute(Grouping.java:370)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:496)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:277)
was:
When we doing a batch of index and search operations to SolrCloud v5.3.2, we
usually met a CPU% spike lasting about 10 min.
You can see the chart in the attach file screenshot-1.png.
During the spike, the thread dump shows most of the threads are with the call
stacks below:
"qtp634210724-4759" #4759 prio=5 os_prio=0 tid=0x00007fb32803e000 nid=0x64e7
runnable [0x00007fb3ef1ef000]
java.lang.Thread.State: RUNNABLE
at
java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444)
at java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419)
at java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298)
at java.lang.ThreadLocal.get(ThreadLocal.java:163)
at
org.apache.solr.search.SolrQueryTimeoutImpl.get(SolrQueryTimeoutImpl.java:49)
at
org.apache.solr.search.SolrQueryTimeoutImpl.shouldExit(SolrQueryTimeoutImpl.java:57)
at
org.apache.lucene.index.ExitableDirectoryReader$ExitableTermsEnum.checkAndThrow(ExitableDirectoryReader.java:165)
at
org.apache.lucene.index.ExitableDirectoryReader$ExitableTermsEnum.<init>(ExitableDirectoryReader.java:157)
at
org.apache.lucene.index.ExitableDirectoryReader$ExitableTerms.iterator(ExitableDirectoryReader.java:141)
at org.apache.lucene.index.TermContext.build(TermContext.java:93)
at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:192)
at
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:855)
at org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:56)
at
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:203)
at
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:855)
at org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:56)
at
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:203)
at
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:855)
at org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:56)
at
org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:203)
at
org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:855)
at
org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:838)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:486)
at
org.apache.solr.search.Grouping.searchWithTimeLimiter(Grouping.java:456)
at org.apache.solr.search.Grouping.execute(Grouping.java:370)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:496)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:277)
> Solr has a CPU% spike when indexing a batch of data
> ---------------------------------------------------
>
> Key: SOLR-9741
> URL: https://issues.apache.org/jira/browse/SOLR-9741
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Affects Versions: 5.3.2
> Environment: Linux 64bit
> Reporter: Forest Soup
> Attachments: screenshot-1.png, threads.zip
>
>
> When we doing a batch of index and search operations to SolrCloud v5.3.2, we
> usually met a CPU% spike lasting about 10 min.
> We have 5 physical servers, 2 solr instances running on each server with
> different port(8983 and 8984), all 8983 are in a same solrcloud, all 8984 are
> in another solrcloud.
> You can see the chart in the attach file screenshot-1.png.
> The thread dump are in the attach file threads.zip.
> During the spike, the thread dump shows most of the threads are with the call
> stacks below:
> "qtp634210724-4759" #4759 prio=5 os_prio=0 tid=0x00007fb32803e000 nid=0x64e7
> runnable [0x00007fb3ef1ef000]
> java.lang.Thread.State: RUNNABLE
> at
> java.lang.ThreadLocal$ThreadLocalMap.getEntryAfterMiss(ThreadLocal.java:444)
> at java.lang.ThreadLocal$ThreadLocalMap.getEntry(ThreadLocal.java:419)
> at
> java.lang.ThreadLocal$ThreadLocalMap.access$000(ThreadLocal.java:298)
> at java.lang.ThreadLocal.get(ThreadLocal.java:163)
> at
> org.apache.solr.search.SolrQueryTimeoutImpl.get(SolrQueryTimeoutImpl.java:49)
> at
> org.apache.solr.search.SolrQueryTimeoutImpl.shouldExit(SolrQueryTimeoutImpl.java:57)
> at
> org.apache.lucene.index.ExitableDirectoryReader$ExitableTermsEnum.checkAndThrow(ExitableDirectoryReader.java:165)
> at
> org.apache.lucene.index.ExitableDirectoryReader$ExitableTermsEnum.<init>(ExitableDirectoryReader.java:157)
> at
> org.apache.lucene.index.ExitableDirectoryReader$ExitableTerms.iterator(ExitableDirectoryReader.java:141)
> at org.apache.lucene.index.TermContext.build(TermContext.java:93)
> at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:192)
> at
> org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:855)
> at
> org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:56)
> at
> org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:203)
> at
> org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:855)
> at
> org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:56)
> at
> org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:203)
> at
> org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:855)
> at
> org.apache.lucene.search.BooleanWeight.<init>(BooleanWeight.java:56)
> at
> org.apache.lucene.search.BooleanQuery.createWeight(BooleanQuery.java:203)
> at
> org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:855)
> at
> org.apache.lucene.search.IndexSearcher.createNormalizedWeight(IndexSearcher.java:838)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:486)
> at
> org.apache.solr.search.Grouping.searchWithTimeLimiter(Grouping.java:456)
> at org.apache.solr.search.Grouping.execute(Grouping.java:370)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:496)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:277)
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
