[
https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229260#comment-15229260
]
Jeff Wartes commented on SOLR-8922:
-----------------------------------
I stumbled onto SJK recently, which provides me a more lightweight way to
measure allocation rate on my production nodes, and also eliminate startup
noise from the measurement.
According to this tool, the node with this patch is allocating heap space at
roughly 60% of the rate that the others are.
That's reasonably consistent with my other measurements, and a pretty big
improvement.
If anyone decides to pull this in, I'd appreciate it getting applied to the 5.5
branch as well, in case there's a 5.5.1 release.
> DocSetCollector can allocate massive garbage on large indexes
> -------------------------------------------------------------
>
> Key: SOLR-8922
> URL: https://issues.apache.org/jira/browse/SOLR-8922
> Project: Solr
> Issue Type: Improvement
> Reporter: Jeff Wartes
> Attachments: SOLR-8922.patch
>
>
> After reaching a point of diminishing returns tuning the GC collector, I
> decided to take a look at where the garbage was coming from. To my surprise,
> it turned out that for my index and query set, almost 60% of the garbage was
> coming from this single line:
> https://github.com/apache/lucene-solr/blob/94c04237cce44cac1e40e1b8b6ee6a6addc001a5/solr/core/src/java/org/apache/solr/search/DocSetCollector.java#L49
> This is due to the simple fact that I have 86M documents in my shards.
> Allocating a scratch array big enough to track a result set 1/64th of my
> index (1.3M) is also almost certainly excessive, considering my 99.9th
> percentile hit count is less than 56k.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
