CachedSQLentity processor is using unbounded hashmap 

                 Key: SOLR-1867
             Project: Solr
          Issue Type: Bug
          Components: contrib - DataImportHandler
    Affects Versions: 1.4
            Reporter: barani

I am using cachedSqlEntityprocessor in DIH to index the data. Please find a 
sample dataconfig structure, 

<entity x query="select * from x"> ---> object 
<entity y query="select * from y" processor="cachedSqlEntityprocessor"> --> object properties 

For each and every object I would be retrieveing corresponding object 
properties (in my subqueries). 

I get in to OOM very often and I think thats a trade off if I use 

My assumption is that when I use cachedSqlEntityprocessor the indexing happens 
as follows, 

First entity x will get executed and the entire table gets stored in cache 
next entity y gets executed and entire table gets stored in cache 
Finally the compasion heppens through hash map . 

So always I need to have the memory allocated to SOLR JVM more than or equal to 
the data present in tables.

One more issue is that even after SOLR completes indexing, the memory used 
previously is not getting released. I could still see the JVM consuming 1.5 GB 
after the indexing completes. I tried to use Java hotspot options but didnt see 
any differences.. GC is not getting invoked even after a long time when using 
CachedSQLentity processor

Main issue seem to be the fact that  the CachedSQLentity processor cache is an 
unbounded HashMap, with no option to bound it. 


This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Reply via email to