Adrien Grand resolved LUCENE-3953.
       Resolution: Fixed
    Fix Version/s: master (7.0)

This looks mostly due to FieldCache storing data in a dense way. Now that 
FieldCache has been replaced with doc values, you would need many more fields 
in order to trigger OOMEs, but disk usage and indexing speed would still suffer 
from these sparse fields. I think we can consider it as fixed in master (7.0) 
thanks to the new sparse APIs for doc values and the new doc values format.

That said, it is still advised to have dense fields whenever possible. We made 
sparse fields more cost-efficient, but there is still a penalty compared to 
dense fields.

> Using many fields in filters, sorts causes OOM
> ----------------------------------------------
>                 Key: LUCENE-3953
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3953
>             Project: Lucene - Core
>          Issue Type: Bug
>          Components: core/search
>    Affects Versions: 4.0-ALPHA
>            Reporter: Des Lownds
>             Fix For: master (7.0)
>         Attachments: FieldCacheLimitPatch.patch
> I have an index with ~1M documents, and many uniquely named fields. Each 
> document only has a few fields populated, but there is a large set of dynamic 
> fields across all docs. In other words, theres many sparse fields.
> Eventually the jvm will fall over due to OOM errors, because lucene's 
> fieldCache grows without bounds. Adding more heap does not help.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to