org.apache.lucene.util.fst.FST taking up lot of Java Heap Memory

[sanjay dutt]

Hello Solr community,
On our Production SolrCloud Server, OutOfMemory has been occurring on lot of 
instances. When I download the HEAP DUMP and analyzed it. I got to know that in 
multiple HEAP DUMPS there are lots of instances of 
org.apache.lucene.codecs.blocktree.BlockTreeTermsReader  which has the highest 
retained heap memory and further I have checked the outgoing-reference for 
those objects, the  org.apache.lucene.util.fst.FST is the one which occupy 90% 
of the heap memory.
it's like 
Production HEAP memory :- 12GBout of which  
org.apache.lucene.codecs.blocktree.BlockTreeTermsReader total retained heap :- 
7-8 GB(vary from instance to instance)and org.apache.lucene.util.fst.FST total 
retained heap :- 6-7 GB
Upon further looking I have calculated the total retained heap for 
FieldReader.fieldInfo.name="my_field" is around 7GB. Now this is the same 
reader which also contains reference to org.apache.lucene.util.fst.FST.
Now "my_field" is the field on which we are performing spatial searches. Is 
spatial searches use FST internally and hence we are seeing lot of heap memory 
used by FST.l only.
IS there any way we can optimize the spatial searches so that it take less 
memory.
Can someone please give me any pointer that from where Should I start looking 
to debug the above issue. 
Thanks and Regards,Sanjay Dutt
Sent from Yahoo Mail on Android

Sort on internal lucene ID led to OOM

[sanjay dutt]

Hello,
We were investigating one HEAP DUMP in which FielsCaceImpl has occupied around 
5GB. Upon further investigation, I got to know that 
FieldCacheImpl$SortedDocValues occupies 90% of the memory where 
FieldCacheImpl$CacheKey is "id".
When I checked the logs I was trying to find any query in which there is any 
sort on id field. But instead I find few queries in which we are sorting on 
_docid_ lucene internal id.
Can sort on _docid_ cause OOM?
And if I will enable docValues for uniqueKey(id) will that solve my problem?
Regards,SanjaySent from Yahoo Mail on Android

Re: org.apache.lucene.util.fst.FST taking up lot of Java Heap Memory

[sanjay dutt]

FieldType defined with class solr.SpatialRecursivePrefixTreeFieldType

In this we are adding points only although collection has few fields with 
points data and then other fieldTypes as well.
And one of the queries looks like 
(my_field: [45,-94 TO 46,-93]+OR+my_field: [42,-94 TO 43,-93])

Thanks and Regards,Sanjay Dutt 

On Thursday, August 6, 2020, 12:10:04 AM GMT+5:30, David Smiley 
 wrote:  
 
 What is the Solr field type definition for this field?  And what sort of
spatial data do you add here -- just points or what?

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Mon, Aug 3, 2020 at 10:09 PM sanjay dutt
 wrote:

> Hello Solr community,
> On our Production SolrCloud Server, OutOfMemory has been occurring on lot
> of instances. When I download the HEAP DUMP and analyzed it. I got to know
> that in multiple HEAP DUMPS there are lots of instances
> of org.apache.lucene.codecs.blocktree.BlockTreeTermsReader  which has the
> highest retained heap memory and further I have checked the
> outgoing-reference for those objects,
> the  org.apache.lucene.util.fst.FST is the one which occupy 90% of the heap
> memory.
> it's like
> Production HEAP memory :- 12GBout of
> which  org.apache.lucene.codecs.blocktree.BlockTreeTermsReader total retained
> heap :- 7-8 GB(vary from instance to
> instance)and org.apache.lucene.util.fst.FST total retained heap :- 6-7 GB
> Upon further looking I have calculated the total retained heap for
> FieldReader.fieldInfo.name="my_field" is around 7GB. Now this is the same
> reader which also contains reference to org.apache.lucene.util.fst.FST.
> Now "my_field" is the field on which we are performing spatial searches.
> Is spatial searches use FST internally and hence we are seeing lot of heap
> memory used by FST.l only.
> IS there any way we can optimize the spatial searches so that it take less
> memory.
> Can someone please give me any pointer that from where Should I start
> looking to debug the above issue.
> Thanks and Regards,Sanjay Dutt
> Sent from Yahoo Mail on Android
  

Re: Solr Down Issue

[sanjay dutt]

It could be OOM killer. Try to monitor it's heap and there is a script in bin 
which basically kills solr when OOM occurs.

Sent from Yahoo Mail on Android 
 
  On Sun, Aug 9, 2020 at 8:14 PM, Ben wrote:   Can you send 
solr logs?

Best,
Ben

On Sun, Aug 9, 2020, 9:55 AM Rashmi Jain  wrote:

> Hello Team,
>
>                I am Rashmi jain implemented solr on one of our site
> bookswagon.com. last 2-3 month we are facing
> strange issue, solr down suddenly without interrupting.  We check solr
> login and also check application logs but no clue found there regarding
> this.
>                We have implemented solr 7.4 on Java SE 10 and have index
> data of books around 28 million.
>                Also we are running solr on Windows server 2012 standard
> with 32 RAM.
>                Please help us on this.
>
> Regards,
> Rashmi
>
>
>
  

Re: org.apache.lucene.util.fst.FST taking up lot of Java Heap Memory

[sanjay dutt]

Best explanation I found so far. Will migrate to LatLonPointSpatialField and 
try to share the benchmark data here. Thanks again David.
Cheers,Sanjay

Sent from Yahoo Mail on Android 
 
  On Sat, Aug 8, 2020 at 3:31 AM, David Smiley wrote:   
Since you have a typical use-case (point data, queries that are
rectangles), I strongly encourage you to migrate to LatLonPointSpatialField:

https://builds.apache.org/job/Solr-reference-guide-master/javadoc/spatial-search.html#latlonpointspatialfield
It's based off an internal "BKD" tree index (doesn't use FSTs) which is
different than the terms based index used by the RPT field that you are
using which employes FSTs.  To be clear, FSTs are awesome but the BKD index
is tailored for numeric data whereas terms/FSTs are not.

If your FSTs are/were taking up so much memory, you are probably not using
Solr 8.4.0 or beyond, which moved to having the FSTs off-heap -- at least
the ones associated with the field indexes.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Thu, Aug 6, 2020 at 8:19 PM sanjay dutt
 wrote:

> FieldType defined with class solr.SpatialRecursivePrefixTreeFieldType
>
> In this we are adding points only although collection has few fields with
> points data and then other fieldTypes as well.
> And one of the queries looks like
> (my_field: [45,-94 TO 46,-93]+OR+my_field: [42,-94 TO 43,-93])
>
> Thanks and Regards,Sanjay Dutt
>
>    On Thursday, August 6, 2020, 12:10:04 AM GMT+5:30, David Smiley <
> dsmi...@apache.org> wrote:
>
>  What is the Solr field type definition for this field?  And what sort of
> spatial data do you add here -- just points or what?
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Aug 3, 2020 at 10:09 PM sanjay dutt
>  wrote:
>
> > Hello Solr community,
> > On our Production SolrCloud Server, OutOfMemory has been occurring on lot
> > of instances. When I download the HEAP DUMP and analyzed it. I got to
> know
> > that in multiple HEAP DUMPS there are lots of instances
> > of org.apache.lucene.codecs.blocktree.BlockTreeTermsReader  which has the
> > highest retained heap memory and further I have checked the
> > outgoing-reference for those objects,
> > the  org.apache.lucene.util.fst.FST is the one which occupy 90% of the
> heap
> > memory.
> > it's like
> > Production HEAP memory :- 12GBout of
> > which  org.apache.lucene.codecs.blocktree.BlockTreeTermsReader total
> retained
> > heap :- 7-8 GB(vary from instance to
> > instance)and org.apache.lucene.util.fst.FST total retained heap :- 6-7 GB
> > Upon further looking I have calculated the total retained heap for
> > FieldReader.fieldInfo.name="my_field" is around 7GB. Now this is the
> same
> > reader which also contains reference to org.apache.lucene.util.fst.FST.
> > Now "my_field" is the field on which we are performing spatial searches.
> > Is spatial searches use FST internally and hence we are seeing lot of
> heap
> > memory used by FST.l only.
> > IS there any way we can optimize the spatial searches so that it take
> less
> > memory.
> > Can someone please give me any pointer that from where Should I start
> > looking to debug the above issue.
> > Thanks and Regards,Sanjay Dutt
> > Sent from Yahoo Mail on Android
>
  

SPLITSHARD failed after running for hours

[sanjay dutt]

Hello Solr community,
We tried to split shard of one collection which contains 80M documents. After 
running for few hours it failed with the exception 
org.apache.solr.common.SolrException.
Upon further investigation, I found below exception
Caused by: java.util.concurrent.RejectedExecutionException: Task 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1@4c995faa 
rejected from 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@4cea9dd8[Terminated,
 pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 
115769]: 

We just ended up with two more sub shards but with ZERO documents in them. Can 
someone please assist me here what exactly may have happened here. And if we 
have to try again what should we keep in our mind so that it will execute 
successfully.
Thanks,Sanjay


Sent from Yahoo Mail on Android

Re: Sort on internal lucene ID led to OOM

[sanjay dutt]

Thanks Erik.When you mentioned we can't control sorting by _docid_ . So if not 
client Does that mean solr internally makes such type of queries which includes 
sort on internal Lucene Id? And for what purpose?
Sent from Yahoo Mail on Android 
 
  On Wed, Aug 5, 2020 at 6:27 PM, Erick Erickson 
wrote:   A sort on anything can cause an OOM… That said, _all_ fields defined 
in your Solr schema should have docValues set to true if you sort, group, use 
function queries or facet on them. What’s happening is the docValues structure 
is being synthesized at runtime on the heap. In recent Solr releases you can 
specify uninvertible=true flag to generate exceptions when this is attempted 
rather than hit an OOM, see: 
https://lucene.apache.org/solr/guide/8_3/defining-fields.html

You can’t control sorting by _docid_, but if you’re specifying that from the 
client, it’s almost certainly not doing what you expect.

Best,
Erick

> On Aug 4, 2020, at 10:19 PM, sanjay dutt  
> wrote:
> 
> Hello,
> We were investigating one HEAP DUMP in which FielsCaceImpl has occupied 
> around 5GB. Upon further investigation, I got to know that 
> FieldCacheImpl$SortedDocValues occupies 90% of the memory where 
> FieldCacheImpl$CacheKey is "id".
> When I checked the logs I was trying to find any query in which there is any 
> sort on id field. But instead I find few queries in which we are sorting on 
> _docid_ lucene internal id.
> Can sort on _docid_ cause OOM?
> And if I will enable docValues for uniqueKey(id) will that solve my problem?
> Regards,SanjaySent from Yahoo Mail on Android
  

Re: _version_ taking too much memory

[sanjay dutt]

Thank you so much Shalin. Yes that would reduce the memory requirement at run 
time.But my next question, Is the "firstSearcher" doing this I mean due to 
which it tries to find the "Max value of version" and for that it it has to do 
the un-inversion. 
Thanks and Regards,Sanjay Dutt 

On Wednesday, November 18, 2020, 12:54:39 PM GMT+5:30, Shalin Shekhar 
Mangar  wrote:  
 
 You should change the _version_ field to be of type docValues and re-index
data. This way you don't have to pay the memory cost of un-inversion.

On Wed, Nov 18, 2020 at 9:51 AM sanjay dutt
 wrote:

> Solr Collection setup
> Shards :- 2Replication :- 4Documents :- 569 Million (I know it's too
> much)Heap Space :- 12GB
> So basically, above collection is having OutOfMemory issues. And upon
> inspection, I got to know
> that org.apache.lucene.uninverting.FieldCacheImpl$LongsFromArray for field
> "_version_" is taking around 50 - 60% memory(5GB).
> Now I am trying to understand How "_version_" internally using the field
> cache?
> Around the same time, In Splunk logs I can see "Found MAX value from Terms
> for _version_ in index". and also few "firstSearcher" queries. Now I think
> upon core initialization "firstSearcher" triggered which then try to find
> the Max value for version using Field Cache but because of too many records
> it occupy lot of memory to carry out the task.
> Can someone please validate Is what I am talking in above para is the
> problem or problem lies in somewhere else?
>
> Best Regards,Sanjay
>


-- 
Regards,
Shalin Shekhar Mangar.
  

_version_ taking too much memory

[sanjay dutt]

Solr Collection setup
Shards :- 2Replication :- 4Documents :- 569 Million (I know it's too much)Heap 
Space :- 12GB
So basically, above collection is having OutOfMemory issues. And upon 
inspection, I got to know that 
org.apache.lucene.uninverting.FieldCacheImpl$LongsFromArray for field 
"_version_" is taking around 50 - 60% memory(5GB).
Now I am trying to understand How "_version_" internally using the field cache?
Around the same time, In Splunk logs I can see "Found MAX value from Terms for 
_version_ in index". and also few "firstSearcher" queries. Now I think upon 
core initialization "firstSearcher" triggered which then try to find the Max 
value for version using Field Cache but because of too many records it occupy 
lot of memory to carry out the task.
Can someone please validate Is what I am talking in above para is the problem 
or problem lies in somewhere else?

Best Regards,Sanjay