Hello,
My ES node is running very high and this is the thread from running
hot_threads. Can someone help me understand why CPU is so high?
This is a 1 node cluster running a linux machine with 16 cores and 72GB of
RAM.
::: [c9q9o027][VTde-tWsTcm9oe7vBZuadA][inet[/10.1.6.21:9300]]
48.3% (241.3ms out of 500ms) cpu usage by thread
'elasticsearch[c9q9o027][search][T#3]'
10/10 snapshots sharing following 17 elements
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.score(ConstantScoreQuery.java:245)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:627)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:119)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:233)
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.lang.Thread.run(Unknown Source)
48.3% (241.3ms out of 500ms) cpu usage by thread
'elasticsearch[c9q9o027][search][T#8]'
2/10 snapshots sharing following 23 elements
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.docID(ConstantScoreQuery.java:190)
org.apache.lucene.search.DisjunctionScorer.heapAdjust(DisjunctionScorer.java:58)
org.apache.lucene.search.DisjunctionSumScorer.nextDoc(DisjunctionSumScorer.java:54)
org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143)
org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134)
org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:69)
org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:119)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:233)
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.lang.Thread.run(Unknown Source)
8/10 snapshots sharing following 23 elements
org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.docID(ConstantScoreQuery.java:190)
org.apache.lucene.search.DisjunctionSumScorer.nextDoc(DisjunctionSumScorer.java:61)
org.apache.lucene.search.DisjunctionSumScorer.nextDoc(DisjunctionSumScorer.java:53)
org.elasticsearch.common.lucene.docset.DocIdSets.toFixedBitSet(DocIdSets.java:143)
org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:134)
org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:69)
org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:119)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:233)
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.lang.Thread.run(Unknown Source)
39.8% (199.2ms out of 500ms) cpu usage by thread
'elasticsearch[c9q9o027][search][T#15]'
10/10 snapshots sharing following 23 elements
org.elasticsearch.common.lucene.search.XBooleanFilter.getDocIdSet(XBooleanFilter.java:224)
org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter.getDocIdSet(ApplyAcceptedDocsFilter.java:45)
org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:128)
org.apache.lucene.search.QueryWrapperFilter$1.iterator(QueryWrapperFilter.java:59)
org.elasticsearch.common.lucene.docset.DocIdSets.toSafeBits(DocIdSets.java:130)
org.elasticsearch.common.lucene.search.FilteredCollector.setNextReader(FilteredCollector.java:69)
org.elasticsearch.common.lucene.MultiCollector.setNextReader(MultiCollector.java:68)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:167)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:119)
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:233)
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:202)
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:203)
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:186)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.lang.Thread.run(Unknown Source)
::: [DeWolff, Jean][Z2ezcYq2QDiEwHjnbsLiYQ][inet[/10.1.6.21:9301]]{client=true,
data=false}
0.9% (4.6ms out of 500ms) cpu usage by thread '<redis'
10/10 snapshots sharing following 35 elements
java.lang.Object.wait(Native Method)
org.jruby.RubyThread.sleep(RubyThread.java:864)
org.jruby.RubyKernel.sleep(RubyKernel.java:801)
org.jruby.RubyKernel$INVOKER$s$0$1$sleep.call(RubyKernel$INVOKER$s$0$1$sleep.gen)
org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:646)
org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:205)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:168)
org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.ast.RescueBodyNode.interpret(RescueBodyNode.java:108)
org.jruby.ast.RescueNode.handleException(RescueNode.java:177)
org.jruby.ast.RescueNode.interpret(RescueNode.java:123)
org.jruby.ast.BeginNode.interpret(BeginNode.java:83)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.WhileNode.interpret(WhileNode.java:131)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:75)
org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:225)
org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:204)
org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:346)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:204)
org.jruby.ast.FCallSpecialArgNode.interpret(FCallSpecialArgNode.java:41)
org.jruby.ast.NewlineNode.interpret(NewlineNode.java:105)
org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:112)
org.jruby.runtime.Interpreted19Block.evalBlockBody(Interpreted19Block.java:209)
org.jruby.runtime.Interpreted19Block.yield(Interpreted19Block.java:197)
org.jruby.runtime.Interpreted19Block.call(Interpreted19Block.java:128)
org.jruby.runtime.Block.call(Block.java:89)
org.jruby.RubyProc.call(RubyProc.java:261)
org.jruby.RubyProc.call(RubyProc.java:213)
org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:98)
java.lang.Thread.run(Unknown Source)
0.0% (127.8micros out of 500ms) cpu usage by thread 'elasticsearch[DeWolff,
Jean][transport_client_timer][T#1]{Hashed wheel timer #1}'
10/10 snapshots sharing following 5 elements
java.lang.Thread.sleep(Native Method)
org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:467)
org.elasticsearch.common.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:376)
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
java.lang.Thread.run(Unknown Source)
0.0% (51.4micros out of 500ms) cpu usage by thread 'JRubyWorker-8'
10/10 snapshots sharing following 9 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(Unknown Source)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown
Source)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown
Source)
java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source)
java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
java.lang.Thread.run(Unknown Source)
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/44e50da7-89b1-415f-ae46-37a317f51e0d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
