On 7/31/2018 2:39 AM, Georg Fette wrote:
We run the server version 7.3.1. on a machine with 32GB RAM in a mode
having -10g.
When requesting a query with
q={!boost
b=sv_int_catalog_count_document}string_catalog_aliases:(*2*)&fq=string_field_type:catalog_entry&rows=2147483647
the server takes all available memory up to 10GB and is then no longer
accessible with one processor at 100%.
When we reduce the rows parameter to 10000000 the query works. The
query returns only 581 results.
This is happening because of the way that Solr prepares for searching.
Objects are allocated in heap memory according to the rows value before
the query even gets executed. If you run Solr on an operating system
other than Windows, the resulting OutOfMemoryError will cause the Solr
process to be killed. If it's running on Windows, Solr would stay
running, but we have no way of knowing whether it would work *correctly*
after OOME.
https://wiki.apache.org/solr/SolrPerformanceProblems#Asking_for_too_many_rows
At the link above is a link to a blog post that covers the problem in
great detail.
https://sbdevel.wordpress.com/2015/10/05/speeding-up-core-search/
With a rows parameter of over 2 billion, Solr (actually it's Lucene,
which provides most of Solr's functionality) will allocate that many
ScoreDoc objects, which needs about 60GB of heap memory. So it's not
possible on your hardware.
As you'll see if you read the blog post, Toke has some ideas about how
to improve the situation. I don't think an issue has been filed, but I
could be wrong about that.
Right now, switching to cursorMark or the /export handler is a better
way to get a very large result set.
Thanks,
Shawn
