and if you provide plenty of memory (8G heap for 111M records with aggregations do not seem enough) for caching of filters and fields and OS memory for caching data files (and/or use SSD) parallel calculation on multiple shards should provide lot better improvement than 50% may be not exactly linear but at least 3-4 times for going from 1 to 6 shards in my opinion) assuming you have more than 6 cores. The memory pressure you mention needs to be removed too. Analyze stats but I suspect 8G is just not enough in your case.
Would be interesting to see if aggregating on rounded (to date) timestamp would improve things on its own. On Sunday, December 7, 2014 5:14:17 AM UTC-5, [email protected] wrote: > > How many docs do you expect your histogram will aggregate? Most of your >> 111M? If so with just one shard and one thread doing the work it is bound >> to be pretty slow. >> > > Expected aggregated records are 78mio. After reindexing with 6 shards per > index the query time reduced by ~50%. The result was surprising: someone > wrote several shards on a single disk have less effect, because they share > the same i/o. But I should mention the threading effect. Are there > recommendations about shard size vs shard count? > > >> Also have you tried moving your not missing filter out of the agg into >> the query filter and also just using > 0 instead of not missing. Also >> reducing precision of the timestamp could possible help > > > Removing the missing filter out of the query gives more speed. I cannot > remember why I used this missing filter. In current test setup the target > result set is identical, even if using 'missing filter'. Is there need to > use 'missing filter' here? What happens, if field 'duration' is missing or > null in some records? > > What is your recommendation to timestamp? Should I replace > > 2014-01-15T14:17:06.245+01:00 > > with less accuracy in minutes > > 2014-01-15T14:17:00.000+01:00 > > ? Would this affect the field data cache? > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b1ce1f3c-f75c-43a9-9eb8-c37116ec2453%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
