On 3/18/16 9:27 AM, Emir Arnautovic wrote:
Running single query that returns all docs and all fields will actually
load as many document as queryResultWindowSize is.
What you need to do is run multiple queries that will return different
documents. In case your id is numeric, you can run something like id:[1
TO 100] and then id:[100 TO 200] etc. Make sure that it is done within
those two minute period if there is any indexing activities.
Would the existing cache be cleared while a active thread is
performing/receiving query?
Your index is relatively small so filter cache of initial size of 1000
entries should take around 20MB (assuming single shard)
Thanks,
Emir
On 18.03.2016 17:02, Rallavagu wrote:
On 3/18/16 8:56 AM, Emir Arnautovic wrote:
Problem starts with autowarmCount="5000" - that executes 5000 queries
when new searcher is created and as queries are executed, document cache
is filled. If you have large queryResultWindowSize and queries return
big number of documents, that will eat up memory before new search is
executed. It probably takes some time as well.
This is also combined with filter cache. How big is your index?
Index is not very large.
numDocs:
85933
maxDoc:
161115
deletedDocs:
75182
Size
1.08 GB
I have run a query to return all documents with all fields. I could
not reproduce OOM. I understand that I need to reduce cache sizes but
wondering what conditions could have caused OOM so I can keep a watch.
Thanks
Thanks,
Emir
On 18.03.2016 15:43, Rallavagu wrote:
Thanks for the recommendations Shawn. Those are the lines I am
thinking as well. I am reviewing application also.
Going with the note on cache invalidation for every two minutes due to
soft commit, wonder how would it go OOM in simply two minutes or is it
likely that a thread is holding the searcher due to long running query
that might be potentially causing OOM? Was trying to reproduce but
could not so far.
Here is the filter cache config
<filterCache class="solr.FastLRUCache" size="5000" initialSize="5000"
autowarmCount="1000"/>
Query Results cache
<queryResultCache class="solr.LRUCache" size="20000"
initialSize="20000" autowarmCount="5000"/>
On 3/18/16 7:31 AM, Shawn Heisey wrote:
On 3/18/2016 8:22 AM, Rallavagu wrote:
So, each soft commit would create a new searcher that would
invalidate
the old cache?
Here is the configuration for Document Cache
<documentCache class="solr.LRUCache" size="100000"
initialSize="100000" autowarmCount="0"/>
<enableLazyFieldLoading>true</enableLazyFieldLoading>
In an earlier message, you indicated you're running into OOM. I think
we can see why with this cache definition.
There are exactly two ways to deal with OOM. One is to increase the
heap size. The other is to reduce the amount of memory that the
program
requires by changing something -- that might be the code, the
config, or
how you're using it.
Start by reducing that cache size to 4096 or 1024.
https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
If yuo've also got a very large filterCache, reduce that size too. The
filterCache typically eats up a LOT of memory, because each entry
in the
cache is very large.
Thanks,
Shawn
