Hello,
I'm using DIH to index the data and the structure of the DIH is like below for
solr core:
<entity>
16 child entities
</entity>
During indexing, since the number of requests being made to database was
high(to process one document 17 queries) and was utilizing most of connections
of database thereby blocking our web application.
To tackle it, we implemented SORTEDMAPBACKEDCACHE with cacheImpl parameter to
reduce the number of requests to database.
<entity name="parententity" pk="PQRS"
query="SELECT
PQRS,PARENT_KEY,L,M,N,O FROM DEF"
>
<field name="L" column="L" />
<field name="M" column="M" />
<field name="N" column="N" />
<entity name="childentity1"
pk="PQRS"
query="SELECT
A,B,C,D,E,F,CHILD_KEY,MODIFY_TS FROM ABC ORDER BY MODIFY_TS DESC"
processor="SqlEntityProcessor" cacheImpl="SortedMapBackedCache"
where="CHILD_KEY=parententity.PARENT_KEY"
>
<field name="A"
column="A" />
<field name="B"
column="B" />
</entity>
.
.
.
.
.
.
.
</entity>
We have 8GB Physical memory system(RAM) with 5GB of it allocated to JVM and
when we do full-import, only 17 requests are made to database. However, it is
shooting up memory consumption and is making the JVM out of memory. Out of
memory is happening depending on the number of records each entity is bringing
in to the memory. For Dev and QA environments, the above memory config is
sufficient. When we move to production, we have to increase the memory to
around 16GB of RAM and 12 GB of JVM.
Is there any logic/configurations to limit the memory usage?
Thanks and Regards,
Srinivas Kashyap
________________________________
DISCLAIMER:
E-mails and attachments from Bamboo Rose, LLC are confidential.
If you are not the intended recipient, please notify the sender immediately by
replying to the e-mail, and then delete it without making copies or using it in
any way.
No representation is made that this email or any attachments are free of
viruses. Virus scanning is recommended and is the responsibility of the
recipient.
