Thanks Jörg. The query does no sorting or scoring. So the child/parent cost must be high -- any ideas on ways to improve look ups -- would docvalues or upgrading to 1.4 help?
Thanks On Saturday, February 28, 2015 at 5:46:59 AM UTC-8, Jörg Prante wrote: > > No idea, except that it could be there is extensive sorting or scoring at > work. This would mean the whole result set has to be iterated/computed to > the point from where it should scroll. Also the parent/child structure > construction is expensive which may be not optimal for scan/scroll as you > already noted. > > Jörg > > On Sat, Feb 28, 2015 at 8:48 AM, Justin Zhu <[email protected] > <javascript:>> wrote: > >> Good question. The request has several filters on child types which may >> have dozens to hundreds of documents per parent document. >> >> It could just be the request is quite complex at calculating the >> documents. The odd factor, is the query is quick at the beginning then >> times out after about 500K documents. >> >> The query fetches 7500 documents each go (15 shards, 500 per shard). >> Typical timing >> 2s >> 3s >> 2s >> ... >> @400K docs scrolled >> 10s >> 12s >> @500K docs scrolled >> timeout (30 seconds) >> ... >> recovers >> 16s -- returns 6500 results instead. >> >> Any idea on why as you scroll deeper, the response time slows any why >> several shards might stop returning results? >> >> >> >> >> On Friday, February 27, 2015 at 8:43:26 AM UTC-8, Jörg Prante wrote: >>> >>> How does the scan/scroll request look like? >>> >>> Jörg >>> >>> On Fri, Feb 27, 2015 at 5:38 PM, Justin Zhu <[email protected]> wrote: >>> >>>> We have a 15 node cluster, 9 data nodes with 18gb ram, 3 master, 3 >>>> client, 12 cpus running 1.3.2. When doing a scroll on an index of 20gb for >>>> 800K documents, each document is 1K we're seeing cpu jump to 30% to 90% on >>>> nodes with primary a shard. Index and other search quests are pretty low. >>>> >>>> Our scroll settings >>>> timeout: 30 seconds >>>> scroll size per shard: 500. >>>> >>>> -- Each scroll request returns 2500 results and takes 10-15 seconds per >>>> request. >>>> - iowait is 0.1% >>>> >>>> Any help appreciated. Details below: >>>> >>>> Hot threads dump >>>> >>>> 95.4% (476.9ms out of 500ms) cpu usage by thread >>>> 'elasticsearch[esd1][search][T#5]' >>>> 5/10 snapshots sharing following 37 elements >>>> org.apache.lucene.util.packed.MonotonicAppendingLongBuffer.get( >>>> MonotonicAppendingLongBuffer.java:70) >>>> org.apache.lucene.util.packed.AbstractAppendingLongBuffer.get( >>>> AbstractAppendingLongBuffer.java:102) >>>> org.apache.lucene.index.XOrdinalMap$2.get(XOrdinalMap.java:260) >>>> org.elasticsearch.index.fielddata.ordinals.GlobalOrdinalMapping. >>>> getGlobalOrd(GlobalOrdinalMapping.java:54) >>>> org.elasticsearch.index.fielddata.ordinals. >>>> GlobalOrdinalMapping.getOrd(GlobalOrdinalMapping.java:63) >>>> org.elasticsearch.index.search.child.ChildrenConstantScoreQuery$ >>>> ParentOrdCollector.collect(ChildrenConstantScoreQuery.java:269) >>>> org.apache.lucene.search.Weight$DefaultBulkScorer. >>>> scoreAll(Weight.java:193) >>>> org.apache.lucene.search.Weight$DefaultBulkScorer. >>>> score(Weight.java:163) >>>> org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) >>>> org.apache.lucene.search.IndexSearcher.search( >>>> IndexSearcher.java:621) >>>> org.apache.lucene.search.IndexSearcher.search( >>>> IndexSearcher.java:309) >>>> org.elasticsearch.index.search.child.ChildrenConstantScoreQuery. >>>> createWeight(ChildrenConstantScoreQuery.java:120) >>>> org.apache.lucene.search.IndexSearcher.createNormalizedWeight( >>>> IndexSearcher.java:684) >>>> org.elasticsearch.search.internal.ContextIndexSearcher. >>>> createNormalizedWeight(ContextIndexSearcher.java:132) >>>> org.elasticsearch.index.search.child.CustomQueryWrappingFilter. >>>> getDocIdSet(CustomQueryWrappingFilter.java:72) >>>> org.elasticsearch.common.lucene.search.XBooleanFilter. >>>> getDocIdSet(XBooleanFilter.java:83) >>>> org.elasticsearch.common.lucene.search.XBooleanFilter. >>>> getDocIdSet(XBooleanFilter.java:83) >>>> org.elasticsearch.common.lucene.search.XBooleanFilter. >>>> getDocIdSet(XBooleanFilter.java:83) >>>> org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter. >>>> getDocIdSet(ApplyAcceptedDocsFilter.java:46) >>>> org.apache.lucene.search.ConstantScoreQuery$ >>>> ConstantWeight.scorer(ConstantScoreQuery.java:157) >>>> org.apache.lucene.search.FilteredQuery$ >>>> RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:547) >>>> org.apache.lucene.search.FilteredQuery$1.scorer( >>>> FilteredQuery.java:136) >>>> org.apache.lucene.search.FilteredQuery$ >>>> RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542) >>>> org.apache.lucene.search.FilteredQuery$FilterStrategy. >>>> filteredBulkScorer(FilteredQuery.java:504) >>>> org.apache.lucene.search.FilteredQuery$1.bulkScorer( >>>> FilteredQuery.java:150) >>>> org.apache.lucene.search.IndexSearcher.search( >>>> IndexSearcher.java:618) >>>> org.elasticsearch.search.internal.ContextIndexSearcher. >>>> search(ContextIndexSearcher.java:175) >>>> org.apache.lucene.search.IndexSearcher.search( >>>> IndexSearcher.java:309) >>>> org.elasticsearch.search.scan.ScanContext.execute( >>>> ScanContext.java:52) >>>> org.elasticsearch.search.query.QueryPhase.execute( >>>> QueryPhase.java:119) >>>> org.elasticsearch.search.SearchService.executeScan( >>>> SearchService.java:237) >>>> org.elasticsearch.search.action.SearchServiceTransportAction$ >>>> SearchScanScrollTransportHandler.messageReceived( >>>> SearchServiceTransportAction.java:856) >>>> org.elasticsearch.search.action.SearchServiceTransportAction$ >>>> SearchScanScrollTransportHandler.messageReceived( >>>> SearchServiceTransportAction.java:845) >>>> org.elasticsearch.transport.netty.MessageChannelHandler$ >>>> RequestHandler.run(MessageChannelHandler.java:275) >>>> java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> ThreadPoolExecutor.java:1145) >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> ThreadPoolExecutor.java:615) >>>> java.lang.Thread.run(Thread.java:744) >>>> 3/10 snapshots sharing following 35 elements >>>> org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc( >>>> FilteredDocIdSetIterator.java:59) >>>> org.apache.lucene.search.FilteredQuery$LeapFrogScorer. >>>> primaryNext(FilteredQuery.java:290) >>>> org.apache.lucene.search.FilteredQuery$ >>>> PrimaryAdvancedLeapFrogScorer.primaryNext(FilteredQuery.java:340) >>>> org.apache.lucene.search.FilteredQuery$LeapFrogScorer. >>>> nextDoc(FilteredQuery.java:285) >>>> org.apache.lucene.search.Weight$DefaultBulkScorer. >>>> scoreAll(Weight.java:192) >>>> org.apache.lucene.search.Weight$DefaultBulkScorer. >>>> score(Weight.java:163) >>>> org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) >>>> org.apache.lucene.search.IndexSearcher.search( >>>> IndexSearcher.java:621) >>>> org.apache.lucene.search.IndexSearcher.search( >>>> IndexSearcher.java:309) >>>> org.elasticsearch.index.search.child.ChildrenConstantScoreQuery. >>>> createWeight(ChildrenConstantScoreQuery.java:120) >>>> org.apache.lucene.search.IndexSearcher.createNormalizedWeight( >>>> IndexSearcher.java:684) >>>> org.elasticsearch.search.internal.ContextIndexSearcher. >>>> createNormalizedWeight(ContextIndexSearcher.java:132) >>>> org.elasticsearch.index.search.child.CustomQueryWrappingFilter. >>>> getDocIdSet(CustomQueryWrappingFilter.java:72) >>>> org.elasticsearch.common.lucene.search.XBooleanFilter. >>>> getDocIdSet(XBooleanFilter.java:83) >>>> org.elasticsearch.common.lucene.search.XBooleanFilter. >>>> getDocIdSet(XBooleanFilter.java:83) >>>> org.elasticsearch.common.lucene.search.XBooleanFilter. >>>> getDocIdSet(XBooleanFilter.java:83) >>>> org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter. >>>> getDocIdSet(ApplyAcceptedDocsFilter.java:46) >>>> org.apache.lucene.search.ConstantScoreQuery$ >>>> ConstantWeight.scorer(ConstantScoreQuery.java:157) >>>> org.apache.lucene.search.FilteredQuery$ >>>> RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:547) >>>> org.apache.lucene.search.FilteredQuery$1.scorer( >>>> FilteredQuery.java:136) >>>> org.apache.lucene.search.FilteredQuery$ >>>> RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542) >>>> org.apache.lucene.search.FilteredQuery$FilterStrategy. >>>> filteredBulkScorer(FilteredQuery.java:504) >>>> org.apache.lucene.search.FilteredQuery$1.bulkScorer( >>>> FilteredQuery.java:150) >>>> org.apache.lucene.search.IndexSearcher.search( >>>> IndexSearcher.java:618) >>>> org.elasticsearch.search.internal.ContextIndexSearcher. >>>> search(ContextIndexSearcher.java:175) >>>> org.apache.lucene.search.IndexSearcher.search( >>>> IndexSearcher.java:309) >>>> org.elasticsearch.search.scan.ScanContext.execute( >>>> ScanContext.java:52) >>>> org.elasticsearch.search.query.QueryPhase.execute( >>>> QueryPhase.java:119) >>>> org.elasticsearch.search.SearchService.executeScan( >>>> SearchService.java:237) >>>> org.elasticsearch.search.action.SearchServiceTransportAction$ >>>> SearchScanScrollTransportHandler.messageReceived( >>>> SearchServiceTransportAction.java:856) >>>> org.elasticsearch.search.action.SearchServiceTransportAction$ >>>> SearchScanScrollTransportHandler.messageReceived( >>>> SearchServiceTransportAction.java:845) >>>> org.elasticsearch.transport.netty.MessageChannelHandler$ >>>> RequestHandler.run(MessageChannelHandler.java:275) >>>> java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> ThreadPoolExecutor.java:1145) >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> ThreadPoolExecutor.java:615) >>>> java.lang.Thread.run(Thread.java:744) >>>> 2/10 snapshots sharing following 30 elements >>>> org.apache.lucene.search.Weight$DefaultBulkScorer. >>>> score(Weight.java:163) >>>> org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) >>>> org.apache.lucene.search.IndexSearcher.search( >>>> IndexSearcher.java:621) >>>> org.apache.lucene.search.IndexSearcher.search( >>>> IndexSearcher.java:309) >>>> org.elasticsearch.index.search.child.ChildrenConstantScoreQuery. >>>> createWeight(ChildrenConstantScoreQuery.java:120) >>>> org.apache.lucene.search.IndexSearcher.createNormalizedWeight( >>>> IndexSearcher.java:684) >>>> org.elasticsearch.search.internal.ContextIndexSearcher. >>>> createNormalizedWeight(ContextIndexSearcher.java:132) >>>> org.elasticsearch.index.search.child.CustomQueryWrappingFilter. >>>> getDocIdSet(CustomQueryWrappingFilter.java:72) >>>> org.elasticsearch.common.lucene.search.XBooleanFilter. >>>> getDocIdSet(XBooleanFilter.java:83) >>>> org.elasticsearch.common.lucene.search.XBooleanFilter. >>>> getDocIdSet(XBooleanFilter.java:83) >>>> org.elasticsearch.common.lucene.search.XBooleanFilter. >>>> getDocIdSet(XBooleanFilter.java:83) >>>> org.elasticsearch.common.lucene.search.ApplyAcceptedDocsFilter. >>>> getDocIdSet(ApplyAcceptedDocsFilter.java:46) >>>> org.apache.lucene.search.ConstantScoreQuery$ >>>> ConstantWeight.scorer(ConstantScoreQuery.java:157) >>>> org.apache.lucene.search.FilteredQuery$ >>>> RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:547) >>>> org.apache.lucene.search.FilteredQuery$1.scorer( >>>> FilteredQuery.java:136) >>>> org.apache.lucene.search.FilteredQuery$ >>>> RandomAccessFilterStrategy.filteredScorer(FilteredQuery.java:542) >>>> org.apache.lucene.search.FilteredQuery$FilterStrategy. >>>> filteredBulkScorer(FilteredQuery.java:504) >>>> org.apache.lucene.search.FilteredQuery$1.bulkScorer( >>>> FilteredQuery.java:150) >>>> org.apache.lucene.search.IndexSearcher.search( >>>> IndexSearcher.java:618) >>>> org.elasticsearch.search.internal.ContextIndexSearcher. >>>> search(ContextIndexSearcher.java:175) >>>> org.apache.lucene.search.IndexSearcher.search( >>>> IndexSearcher.java:309) >>>> org.elasticsearch.search.scan.ScanContext.execute( >>>> ScanContext.java:52) >>>> org.elasticsearch.search.query.QueryPhase.execute( >>>> QueryPhase.java:119) >>>> org.elasticsearch.search.SearchService.executeScan( >>>> SearchService.java:237) >>>> org.elasticsearch.search.action.SearchServiceTransportAction$ >>>> SearchScanScrollTransportHandler.messageReceived( >>>> SearchServiceTransportAction.java:856) >>>> org.elasticsearch.search.action.SearchServiceTransportAction$ >>>> SearchScanScrollTransportHandler.messageReceived( >>>> SearchServiceTransportAction.java:845) >>>> org.elasticsearch.transport.netty.MessageChannelHandler$ >>>> RequestHandler.run(MessageChannelHandler.java:275) >>>> java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> ThreadPoolExecutor.java:1145) >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> ThreadPoolExecutor.java:615) >>>> java.lang.Thread.run(Thread.java:744) >>>> >>>> 58.8% (294.1ms out of 500ms) cpu usage by thread >>>> 'elasticsearch[esd1][search][T#3]' >>>> 3/10 snapshots sharing following 19 elements >>>> org.apache.lucene.search.FilteredDocIdSetIterator.nextDoc( >>>> FilteredDocIdSetIterator.java:60) >>>> org.apache.lucene.search.ConstantScoreQuery$ >>>> ConstantScorer.nextDoc(ConstantScoreQuery.java:257) >>>> org.apache.lucene.search.Weight$DefaultBulkScorer. >>>> scoreAll(Weight.java:192) >>>> org.apache.lucene.search.Weight$DefaultBulkScorer. >>>> score(Weight.java:163) >>>> org.apache.lucene.search.BulkScorer.score(BulkScorer.java:35) >>>> org.apache.lucene.search.IndexSearcher.search( >>>> IndexSearcher.java:621) >>>> org.elasticsearch.search.internal.ContextIndexSearcher. >>>> search(ContextIndexSearcher.java:175) >>>> org.apache.lucene.search.IndexSearcher.search( >>>> IndexSearcher.java:581) >>>> org.apache.lucene.search.IndexSearcher.search( >>>> IndexSearcher.java:533) >>>> org.apache.lucene.search.IndexSearcher.search( >>>> IndexSearcher.java:510) >>>> org.apache.lucene.search.IndexSearcher.search( >>>> IndexSearcher.java:345) >>>> org.elasticsearch.search.query.QueryPhase.execute( >>>> QueryPhase.java:149) >>>> org.elasticsearch.search.SearchService.executeQueryPhase( >>>> SearchService.java:261) >>>> org.elasticsearch.search.action.SearchServiceTransportAction$ >>>> SearchQueryTransportHandler.messageReceived( >>>> SearchServiceTransportAction.java:688) >>>> org.elasticsearch.search.action.SearchServiceTransportAction$ >>>> SearchQueryTransportHandler.messageReceived( >>>> SearchServiceTransportAction.java:677) >>>> org.elasticsearch.transport.netty.MessageChannelHandler$ >>>> RequestHandler.run(MessageChannelHandler.java:275) >>>> java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> ThreadPoolExecutor.java:1145) >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> ThreadPoolExecutor.java:615) >>>> java.lang.Thread.run(Thread.java:744) >>>> 7/10 snapshots sharing following 10 elements >>>> sun.misc.Unsafe.park(Native Method) >>>> java.util.concurrent.locks.LockSupport.park(LockSupport. >>>> java:186) >>>> java.util.concurrent.LinkedTransferQueue.awaitMatch( >>>> LinkedTransferQueue.java:735) >>>> java.util.concurrent.LinkedTransferQueue.xfer( >>>> LinkedTransferQueue.java:644) >>>> java.util.concurrent.LinkedTransferQueue.take( >>>> LinkedTransferQueue.java:1137) >>>> org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take( >>>> SizeBlockingQueue.java:162) >>>> java.util.concurrent.ThreadPoolExecutor.getTask( >>>> ThreadPoolExecutor.java:1068) >>>> java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> ThreadPoolExecutor.java:1130) >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> ThreadPoolExecutor.java:615) >>>> java.lang.Thread.run(Thread.java:744) >>>> >>>> 41.0% (204.8ms out of 500ms) cpu usage by thread >>>> 'elasticsearch[esd1][search][T#6]' >>>> 10/10 snapshots sharing following 10 elements >>>> sun.misc.Unsafe.park(Native Method) >>>> java.util.concurrent.locks.LockSupport.park(LockSupport. >>>> java:186) >>>> java.util.concurrent.LinkedTransferQueue.awaitMatch( >>>> LinkedTransferQueue.java:735) >>>> java.util.concurrent.LinkedTransferQueue.xfer( >>>> LinkedTransferQueue.java:644) >>>> java.util.concurrent.LinkedTransferQueue.take( >>>> LinkedTransferQueue.java:1137) >>>> org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take( >>>> SizeBlockingQueue.java:162) >>>> java.util.concurrent.ThreadPoolExecutor.getTask( >>>> ThreadPoolExecutor.java:1068) >>>> java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> ThreadPoolExecutor.java:1130) >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> ThreadPoolExecutor.java:615) >>>> java.lang.Thread.run(Thread.java:744) >>>> >>>> >>>> Bigdesk graphs >>>> >>>> >>>> <https://lh5.googleusercontent.com/-czxx8fTd_Oo/VPCcfre2mcI/AAAAAAAAAe8/v3fk9toDaF8/s1600/Bigdesk.png> >>>> \ >>>> >>>> >>>> <https://lh4.googleusercontent.com/-au5U3a3Qjew/VPCcztClm_I/AAAAAAAAAfE/o5qnPrTYRxQ/s1600/Bigdesk.png> >>>> >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit https://groups.google.com/d/ >>>> msgid/elasticsearch/72fb97ee-a02f-48e3-b217-cd3e15cc24e3% >>>> 40googlegroups.com >>>> <https://groups.google.com/d/msgid/elasticsearch/72fb97ee-a02f-48e3-b217-cd3e15cc24e3%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/9f8a64a8-5711-48d0-b2ad-d3472f442935%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f0e8a4ad-9d5a-42d8-b69e-fb4ccaf75c26%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
