Thats a lot of threads for 1G of memory. On Friday, April 3, 2015 at 4:55:20 AM UTC-6, cyrilforce wrote: > > Hi, > > Having to test the performance of the ES with some load testing it > reaches 100% CPU utilization and following is hot threads : > > ES configuration > --------------------------------- > ES_MIN_MEM 256m > ES_MAX_MEM 1g > > > Load testing configuration > -------------------------------------------------- > From 100 threads per second ramping up to 400 threads per second over > period of 60 second and it starts failing we reaches 300+ threads per > seconds > > > Hot threads > ------------------- > :/elasticsearch/logs ]$ curl -XGET ' > http://localhost:9200//_nodes/hot_threads?pretty' > :: [Mesmero][if9uerQWRVGmpD3iq0ELYg][2905f3223ee8][inet[/172.17.0.62:9300 > ]] > > 37.6% (188ms out of 500ms) cpu usage by thread > 'elasticsearch[Mesmero][search][T#1]' > 2/10 snapshots sharing following 6 elements > java.lang.System.currentTimeMillis(Native Method) > > org.elasticsearch.action.search.type.AbstractAsyncAction.buildTookInMillis(AbstractAsyncAction.java:46) > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:150 > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > 4/10 snapshots sharing following 6 elements > > org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231) > > org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228) > > org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > 3/10 snapshots sharing following 18 elements > > org.apache.lucene.util.AttributeSource.clearAttributes(AttributeSource.java:279) > > org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:194) > > org.apache.lucene.analysis.standard.StandardFilter.incrementToken(StandardFilter.java:57) > > org.apache.lucene.analysis.en.EnglishPossessiveFilter.incrementToken(EnglishPossessiveFilter.java:57) > > org.apache.lucene.analysis.core.LowerCaseFilter.incrementToken(LowerCaseFilter.java:62) > > org.apache.lucene.analysis.util.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:90) > > org.apache.lucene.analysis.snowball.SnowballFilter.incrementToken(SnowballFilter.java:91) > > org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:218) > > org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118) > > org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128) > > org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211) > > org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481) > > org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452) > > org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449) > > org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > unique snapshot > sun.nio.ch.NativeThread.current(Native Method) > sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46) > sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:695) > sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:684) > > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:179) > > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:342) > > org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:140) > > org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116) > org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:101) > > org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:135) > > org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:354) > > org.apache.lucene.index.SegmentReader.document(SegmentReader.java:335) > > org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:110) > org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:196) > > org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:228) > > org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:156) > > org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481) > > org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452) > > org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449) > > org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > 34.0% (169.9ms out of 500ms) cpu usage by thread > 'elasticsearch[Mesmero][search][T#3]' > 2/10 snapshots sharing following 13 elements > java.util.IdentityHashMap.entrySet(IdentityHashMap.java:1098) > > org.elasticsearch.search.internal.InternalSearchHits.writeTo(InternalSearchHits.java:251) > > org.elasticsearch.search.internal.InternalSearchHits.writeTo(InternalSearchHits.java:230) > > org.elasticsearch.search.internal.InternalSearchResponse.writeTo(InternalSearchResponse.java:138) > > org.elasticsearch.action.search.SearchResponse.writeTo(SearchResponse.java:262) > > org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:91) > > org.elasticsearch.transport.netty.NettyTransportChannel.sendResponse(NettyTransportChannel.java:68) > > org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:67) > > org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:63) > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:150 > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > 3/10 snapshots sharing following 20 elements > > org.elasticsearch.index.query.ScriptFilterParser$ScriptFilter$ScriptDocSet.matchDoc(ScriptFilterParser.java:190) > > org.elasticsearch.common.lucene.docset.MatchDocIdSet.get(MatchDocIdSet.java:67) > > org.elasticsearch.common.lucene.search.FilteredCollector.collect(FilteredCollector.java:60) > > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193) > > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) > > org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269) > > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157) > > org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:275) > > org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231) > > org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228) > > org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > 3/10 snapshots sharing following 12 elements > > org.apache.lucene.analysis.snowball.SnowballFilter.incrementToken(SnowballFilter.java:96) > > org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:218) > > org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118) > > org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128) > > org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211) > > org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481) > > org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452) > > org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449) > > org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > 2/10 snapshots sharing following 10 elements > > org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:118) > > org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:128) > > org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211) > > org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481) > > org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452) > > org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449) > > org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > > 22.8% (114.1ms out of 500ms) cpu usage by thread > 'elasticsearch[Mesmero][search][T#4]' > 5/10 snapshots sharing following 36 elements > java.lang.Throwable.fillInStackTrace(Native Method) > java.lang.Throwable.fillInStackTrace(Throwable.java:783) > java.lang.Throwable.<init>(Throwable.java:250) > java.lang.Exception.<init>(Exception.java:54) > java.lang.RuntimeException.<init>(RuntimeException.java:51) > > org.antlr.v4.runtime.RecognitionException.<init>(RecognitionException.java:65) > > org.antlr.v4.runtime.NoViableAltException.<init>(NoViableAltException.java:70) > > org.antlr.v4.runtime.atn.ParserATNSimulator.noViableAlt(ParserATNSimulator.java:1776) > > org.antlr.v4.runtime.atn.ParserATNSimulator.execATN(ParserATNSimulator.java:496) > > org.antlr.v4.runtime.atn.ParserATNSimulator.adaptivePredict(ParserATNSimulator.java:423) > > com.test.SecurityExpressionParser.security_rule(SecurityExpressionParser.java:284) > > com.test.SecurityExpressionParser.security_rule(SecurityExpressionParser.java:310) > > com.test.SecurityExpressionParser.security_rule(SecurityExpressionParser.java:227) > com.test.ExpressionManager.compile(ExpressionManager.java:134) > > com.test.ExpressionManager.valiateExpression(ExpressionManager.java:293) > com.test.VisibilityFilter.run(VisibilityFilter.java:107) > > org.elasticsearch.index.query.ScriptFilterParser$ScriptFilter$ScriptDocSet.matchDoc(ScriptFilterParser.java:190) > > org.elasticsearch.common.lucene.docset.MatchDocIdSet.get(MatchDocIdSet.java:67) > > org.elasticsearch.common.lucene.search.FilteredCollector.collect(FilteredCollector.java:60) > > org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:193) > > org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:163) > org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) > > org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:191) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:491) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:448) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:281) > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:269) > > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:157) > > org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:275) > > org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231) > > org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228) > > org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > 4/10 snapshots sharing following 7 elements > > org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481) > > org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452) > > org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449) > > org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > java.lang.Thread.run(Thread.java:745) > unique snapshot > java.lang.System.nanoTime(Native Method) > > org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:487) > > org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452) > > org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449) > > org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559) > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > Anyone have any idea ? > > > > > > >
-- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c9abadde-e489-467c-ada7-062bd7463ea2%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
