So does that mean all the documents in the shard (since the index has only 1 shard) are pulled into memory and then the filter is applied?
By moving it to a post-filter, I see the response coming back in around 15s than previously where it was taking more than a minute. However the memory still increases around 2-3GB. Re-running the filtered query again multiple times does not further increase the memory. Though the index store mentions only 504mb could you explain why the memory spikes to 2-3GB even with a filtered-query? With the filtered query approach does the filtering happen at the disk level? Could you also explain why I don't see the memory increasing further with multiple runs of the filtered-query? On Friday, November 21, 2014 6:15:27 PM UTC+5:30, Nick Canzoneri wrote: > > This is something that I just "discovered" as well. > > Using a top-level filter is really a "post_filter" (it's renamed in later > versions of ES): > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-post-filter.html > > So, this will execute the query first (a default "match_all: {}") and then > execute the filter on that result set. This is not very efficient for your > query, since I expect you expected having filter there to work like a > "pre-filter" and filter out results *before* executing the query. > > To do that, you need to use a "filtered query": > http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-filtered-query.html > > In your case, the resulting query would look like: > > curl -XGET 'http://localhost:9200/my-index/my-doc-type/_search' -d '{ > "query": { > "filtered": { > "filter": { > "term": {"void": false} > } > } > }, > "fields": [["user_id1", "user_name", "date", "status", "q1", > "q1_unique_code", "q2", "q3"]], > "size": 50000, "sort": ["date_value"]}' > > > > On Fri, Nov 21, 2014 at 7:07 AM, Ajay Divakaran <[email protected] > <javascript:>> wrote: > >> The term filter that is used: >> >> curl -XGET 'http://localhost:9200/my-index/my-doc-type/_search' -d '{ >> "filter": { >> "term": {"void": false} >> }, >> "fields": [["user_id1", "user_name", "date", "status", "q1", >> "q1_unique_code", "q2", "q3"]], >> "size": 50000, "sort": ["date_value"]}' >> >> >> - The 'void' field is a boolean field. >> - The index store size is 504mb. >> - The elastic search setup consists of only a single node and the >> index consists of only a single shard and 0 replicas. The version of >> elasticsearch is 0.90.7 >> - The fields mentioned above is only the first 8 fields. The actual >> term filter that we execute has 350 fields mentioned. >> >> *We noticed the memory spiking by about 2-3gb though the store size is >> only 504mb.* >> >> *Running the query multiple times seems to continuously increase the >> memory.* >> >> Could someone explain why this memory spike occurs? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/7c4ea660-9411-4d1d-a86c-84f1c43f4f7e%40googlegroups.com >> >> <https://groups.google.com/d/msgid/elasticsearch/7c4ea660-9411-4d1d-a86c-84f1c43f4f7e%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Nick Canzoneri > Developer, Wildbit <http://wildbit.com/> > Beanstalk <http://beanstalkapp.com/>, Postmark <http://postmarkapp.com/>, > dploy.io > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d6436153-49b1-4a07-ab57-d135e035f84d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
