[
https://issues.apache.org/jira/browse/LUCENE-6842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14963326#comment-14963326
]
Ryan Josal commented on LUCENE-6842:
------------------------------------
FWIW, I have seen an OOME due to a large number of fields before. I had an
index with millions of documents and I wanted to experiment with ways to reduce
full indexing time, and there was a way to reorganize the data that would
reduce the number of docs by a factor of 2000 by adding 2000 fields (using a
solr dynamic multivalued String with ~2-3 values on average). The idea was
quickly canned after seeing an OOME during full indexing. This piece of data
we may eventually move out of Lucene and into a datastore more suited like
Cassandra.
> No way to limit the fields cached in memory and leads to OOM when there are
> thousand of fields (thousands)
> ----------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-6842
> URL: https://issues.apache.org/jira/browse/LUCENE-6842
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
> Affects Versions: 4.6.1
> Environment: Linux, openjdk 1.6.x
> Reporter: Bala Kolla
> Attachments: HistogramOfHeapUsage.png
>
>
> I am opening this defect to get some guidance on how to handle a case of
> server running out of memory and it seems like it's something to do how we
> index. But want to know if there is anyway to reduce the impact of this on
> memory usage before we look into the way of reducing the number of fields.
> Basically we have many thousands of fields being indexed and it's causing a
> large amount of memory being used (25GB) and eventually leading to
> application to hang and force us to restart every few minutes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
