Hi Thomas Corthals, After experimenting with a page size of 100,000 records, I successfully retrieved approximately 19 million records and saved the response to a PDF file from our application. However, I noticed that the Solr heap memory did not release after completing the activity. Subsequent attempts to perform the same activity resulted in the heap memory being occupied again. With repeated iterations, the heap memory reached 100%, causing Solr to become unresponsive.
In an attempt to mitigate this issue, I adjusted the filterCache parameter configuration size from the default value of 512 to 50 in the solrconfig.xml file. Although this adjustment did not significantly impact performance, I observed only a slight variation in memory usage. I have attached screenshots and metrics for reference. Could you please provide guidance on how to address this issue? Below screenshot is before running the activity [image: image.png] Below screenshot is after completion of the activity [image: image.png] On Fri, Apr 5, 2024 at 4:03 PM Thomas Corthals <tho...@klascement.net> wrote: > Hi Prasad, > > Have you tried with a smaller page size? Just how many documents you can > fetch in one page with the given memory depends on the size of the > documents. You'll have to try out what works for you. > > Regardless of page size a cursor will still be the way to go to page > through a large set of documents. > > Thomas > > Op vr 5 apr 2024 om 12:10 schreef prasad bezavada < > prasadbezav...@gmail.com > >: > > > Hello Thomas Corthals, > > > > Thank you very much for your valuable reply. I am trying to use cursors, > > but for the first query also its taking so much time to get the results, > > and next query I am getting heap memory error in my java application. > > > > On Fri, Apr 5, 2024 at 2:41 PM Thomas Corthals <tho...@klascement.net> > > wrote: > > > > > Hi Prasad, > > > > > > This is expected with "deep paging": > > > > > > > > > https://solr.apache.org/guide/solr/latest/query-guide/pagination-of-results.html#performance-problems-with-deep-paging > > > > > > Have a look at cursors instead, that should solve your problem: > > > > > > > > > https://solr.apache.org/guide/solr/latest/query-guide/pagination-of-results.html#fetching-a-large-number-of-sorted-results-cursors > > > > > > Thomas > > > > > > Op vr 5 apr 2024 om 10:36 schreef prasad bezavada < > > > prasadbezav...@gmail.com > > > >: > > > > > > > Dear Team, > > > > > > > > I'm currently using Solr version 8.11.3, configured with RAM > resources > > > (125 > > > > GB physical memory, 64 GB heap memory). The collection comprises 4 > > shards > > > > within the same node. Through our Java application ( SolrJ), > > > > indexed approximately 8 million records from an RDBMS table into > Solr. > > > > > > > > Presently, my task is to query this indexes and exporting the results > > (5 > > > > million records fetched with my solr query) to PDF format via our > Java > > > > application. To avoid potential heap memory issues, I've implemented > > > > pagination (3 lakhs) in the query using start and setrows parameters. > > > > > > > > However, I've encountered an issue where the response time for > > subsequent > > > > queries to fetch the next set of results (e.g., 3 to 6 lakhs, 6 to 9 > > > lakhs) > > > > progressively increases, leading to socket timeout exceptions. > > > > Additionally, Solr's physical memory consumption exceeds 90%, without > > > > releasing it. > > > > > > > > I have several queries regarding this situation: > > > > > > > > Why does the query time in Solr increase with each pagination query? > > > > What causes Solr to occupy over 90% of physical memory and fail to > > > release > > > > it? > > > > What would be the optimal approach for retrieving 5 million records > > from > > > > our Java application and exporting them to PDF or other file formats? > > > > Your insights and suggestions on resolving these issues would be > > greatly > > > > appreciated. > > > > > > > > > > > > -- > > > > *Thanks&Regards* > > > > > > > > *Prasad Bezavada* > > > > > > > > > > > > > -- > > *Thanks&Regards* > > > > *Prasad Bezavada* > > > -- *Thanks&Regards* *Prasad Bezavada*
