https://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_a_MySQL_database._My_table_is_huge_and_DataImportHandler_is_going_out_of_memory._Why_does_DataImportHandler_bring_everything_to_memory.3F
-Original Message-
From: Deeksha Sharma [mailto:dsha...@flexera.com]
Sent: Wednesday, September 27, 2017 1:40 PM
To: solr-user@lucene.apache.org
Subject: DataImport Handler Out of Memory
I am trying to create indexes using dataimport handler (Solr 5.2.1). Data is in
mysql db and the number of records are more than 3.5 million. My solr server
stops due to OOM (out of memory error). I tried starting solr by giving 12GB of
RAM but still no luck.
Also, I see that Solr fetches all the documents in 1 request. Is there a way to
configure Solr to stream the data from DB or any other solution somewhere may
have tried?
Note: When my records are nearly 2 Million, I am able to create indexes by
giving Solr 10GB of RAM.
Your help is appreciated.
Thanks
Deeksha